Instance Meta request timeout

aronglia commented 4 years ago

Do not use the issues tracker for help or support (try Elixir Forum, Slack, IRC, etc.)
Questions about how to contribute are fine.

Environment

Elixir & Erlang versions (elixir --version): erlang 22.0 elixir 1.9.0
ExAws version mix deps |grep ex_aws 2.1.1 1e4de2106cfbf4e837de41be41cd15813eabc722315e388f0d6bb3732cec47cd
HTTP client version. IE for hackney do mix deps | grep hackney hackney 1.15.1 9f8f471c

Current behavior

From time to time the request to the Instance Metadata fails with this error message:

RuntimeError: Instance Meta Error: {:error, %{reason: :timeout}}

You tried to access the AWS EC2 instance meta, but it could not be reached.
This happens most often when trying to access it from your local computer,
which happens when environment variables are not set correctly prompting
ExAws to fallback to the Instance Meta.

Please check your key config and make sure they're configured correctly:

For Example:
ExAws.Config.new(:s3)
ExAws.Config.new(:dynamodb)

  File "lib/ex_aws/instance_meta.ex", line 29, in ExAws.InstanceMeta.request/2
  File "lib/ex_aws/instance_meta.ex", line 66, in ExAws.InstanceMeta.instance_role_credentials/1
  File "lib/ex_aws/instance_meta.ex", line 74, in ExAws.InstanceMeta.security_credentials/1
  File "lib/ex_aws/config/auth_cache.ex", line 79, in ExAws.Config.AuthCache.refresh_config/2
  File "lib/ex_aws/config/auth_cache.ex", line 51, in ExAws.Config.AuthCache.handle_info/2
  File "gen_server.erl", line 637, in :gen_server.try_dispatch/4
  File "gen_server.erl", line 711, in :gen_server.handle_msg/6
  File "proc_lib.erl", line 249, in :proc_lib.init_p_do_apply/3

I think this occurs when AWS metadata service is down or there is a network error. I think there should be retries for this request.

Expected behavior

The request to instance metadata should be retried.

benwilson512 commented 4 years ago

Are you running on Kubernetes or a similar container management scheme?

aronglia commented 4 years ago

Are you running on Kubernetes or a similar container management scheme?

Yes, I am running on Kubernetes.

peillis commented 3 years ago

I've seen the same error, but not using containers, just regular ec2 instances.

ex-aws / ex_aws