Open warmwaffles opened 2 months ago
I would love to have this functionality. Would you be willing to take a stab at this?
Cool, just wanted to make sure it is desired. I have some of this done already. I just need to finish working it out and getting tests written for it.
@rzane here's the current implementation I have cooked up.
It's built on top of #32. The problem with Elixir 1.16 is the soft deprecation of File.stream!/3
. Once that one is merged, I'll open a PR with the stream interface added and we can work through it some.
I am running into issues at the moment with minio locally and I need to play with my configurations first. Here's the error:
** (exit) exited in: GenServer.call(ExAws.Config.AuthCache, {:refresh_auth, %{port: 443, scheme: "https://", host: "s3.amazonaws.com", http_client: ExAws.Request.Hackney, region: "us-east-1", access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role], secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role], json_codec: Jason, retries: [max_attempts: 10, base_backoff_in_ms: 10, max_backoff_in_ms: 10000], normalize_path: true, require_imds_v2: false}}, 30000)
** (EXIT) an exception was raised:
** (RuntimeError) Instance Meta Error: {:error, %{reason: :connect_timeout}}
You tried to access the AWS EC2 instance meta, but it could not be reached.
This happens most often when trying to access it from your local computer,
which happens when environment variables are not set correctly prompting
ExAws to fallback to the Instance Meta.
Please check your key config and make sure they're configured correctly:
For Example:
ExAws.Config.new(:s3)
ExAws.Config.new(:dynamodb)
```
(ex_aws 2.5.3) lib/ex_aws/instance_meta.ex:27: ExAws.InstanceMeta.request/3
(ex_aws 2.5.3) lib/ex_aws/instance_meta.ex:84: ExAws.InstanceMeta.instance_role_credentials/1
(ex_aws 2.5.3) lib/ex_aws/instance_meta.ex:92: ExAws.InstanceMeta.security_credentials/1
(ex_aws 2.5.3) lib/ex_aws/config/auth_cache.ex:132: ExAws.Config.AuthCache.refresh_auth_now/2
(ex_aws 2.5.3) lib/ex_aws/config/auth_cache.ex:45: ExAws.Config.AuthCache.handle_call/3
(stdlib 5.2) gen_server.erl:1131: :gen_server.try_handle_call/4
(stdlib 5.2) gen_server.erl:1160: :gen_server.handle_msg/6
(stdlib 5.2) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
I have been thinking about this some as I have a need to be able to stream a file from an S3 like resource or local disk. I wanted to know your thoughts on this @rzane
Adding
For the null adapter this implementation is dead simple
For the Disk adapter it's just as trivial
For S3 it's a little trickier but the implementation is essentially spelled out in the documentation here https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#download_file/4