search

ex-aws / ex_aws

A flexible, easy to use set of clients AWS APIs for Elixir
https://hex.pm/packages/ex_aws
MIT License
1.27k stars 526 forks source link

Latest version crashes on ECS fargate #869

Closed hoyon closed 2 years ago

hoyon commented 2 years ago

Environment

Current behavior

Running an elixir app with ex_aws on ECS fargate crashes on startup when using the latest version of ex_aws.

Error message:

GenServer ExAws.InstanceMetaTokenProvider terminating
** (RuntimeError) Instance Meta Error: {:error, %{reason: :checkout_failure}}

You tried to access the AWS EC2 Instance Metadata token API, but it could not be reached.

Please check AWS EC2 Instance Metadata Service configuration to make sure the service is enabled.

    (ex_aws 2.3.0) lib/ex_aws/instance_meta_token_provider.ex:94: ExAws.InstanceMetaTokenProvider.request_token/1
    (ex_aws 2.3.0) lib/ex_aws/instance_meta_token_provider.ex:57: ExAws.InstanceMetaTokenProvider.refresh_token/2
    (ex_aws 2.3.0) lib/ex_aws/instance_meta_token_provider.ex:47: ExAws.InstanceMetaTokenProvider.handle_call/3
    (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4
    (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6
    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message (from ExAws.Config.AuthCache): {:refresh_token, %{access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role], host: "s3.eu-west-2.amazonaws.com", http_client: ExAws.Request.Hackney, json_codec: Jason, normalize_path: true, port: 443, region: "eu-west-2", retries: [max_attempts: 10, base_backoff_in_ms: 10, max_backoff_in_ms: 10000], scheme: "https://", secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]}}

I think this is because of the new IMDSv2 support added in this pr which crashes as the IMDSv2 endpoint isn't available on ECS fargate.

This command from the docs doesn't work on ECS fargate for example: 

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`

Instead AWS_CONTAINER_CREDENTIALS_RELATIVE_URI should be used as in ExAws.InstanceMeta

Expected behavior

The app starts successfully without error and credentials are fetched using the existing AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable and doesn't attempt to use IMDSv2.

RobinFrcd commented 2 years ago

If you guys have the same issue with ECS on EC2, here's a way to fix this (with IMDSv2 best practices):

resource "aws_instance"  ... {
  metadata_options {
    http_tokens                 = "required"
    http_endpoint               = "enabled"
    http_put_response_hop_limit = 2 # https://stackoverflow.com/a/71884476/5692012
  }
}