Incorrect AWS metadata server path when running on Fargate

jonathanasdf commented 2 years ago

Runs fine on EC2, but running on Fargate gives

    _gs.client = storage.Client()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/storage/client.py", line 161, in __init__
    super(Client, self).__init__(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/client.py", line 319, in __init__
    Client.__init__(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/client.py", line 178, in __init__
    credentials, _ = google.auth.default(scopes=scopes)
  File "/usr/local/lib/python3.9/site-packages/google/auth/_default.py", line 473, in default
    project_id = credentials.get_project_id(request=request)
  File "/usr/local/lib/python3.9/site-packages/google/auth/external_account.py", line 269, in get_project_id
    self.before_request(request, "GET", url, headers)
  File "/usr/local/lib/python3.9/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.9/site-packages/google/auth/external_account.py", line 290, in refresh
    self._impersonated_credentials.refresh(request)
  File "/usr/local/lib/python3.9/site-packages/google/auth/impersonated_credentials.py", line 242, in refresh
    self._update_token(request)
  File "/usr/local/lib/python3.9/site-packages/google/auth/impersonated_credentials.py", line 255, in _update_token
    self._source_credentials.refresh(request)
  File "/usr/local/lib/python3.9/site-packages/google/auth/external_account.py", line 298, in refresh
    subject_token=self.retrieve_subject_token(request),
  File "/usr/local/lib/python3.9/site-packages/google/auth/aws.py", line 461, in retrieve_subject_token
    aws_security_credentials = self._get_security_credentials(request)
  File "/usr/local/lib/python3.9/site-packages/google/auth/aws.py", line 586, in _get_security_credentials
    role_name = self._get_metadata_role_name(request)
  File "/usr/local/lib/python3.9/site-packages/google/auth/aws.py", line 660, in _get_metadata_role_name
    response = request(url=self._security_credentials_url, method="GET")
  File "/usr/local/lib/python3.9/site-packages/google/auth/transport/requests.py", line 189, in __call__
    six.raise_from(new_exc, caught_exc)
  File "<string>", line 3, in raise_from
google.auth.exceptions.TransportError: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/meta-data/iam/security-credentials (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f7839bd5280>, 'Connection to 169.254.169.254 timed out. (connect timeout=120)'))

Looking at https://stackoverflow.com/questions/57065458/cannot-access-instance-metadata-from-within-a-fargate-task it appears fargate has a different endpoint for iam metadata

tomwphillips commented 2 years ago

I encountered the same problem. As a workaround, I used boto3 to get credentials and then set the environment variables manually. The Google auth library will use the environment variables instead of querying the EC2 metadata service.

import os

import boto3
from google.auth import environment_vars

aws_credentials = boto3.Session().get_credentials().get_frozen_credentials()
os.environ[environment_vars.AWS_ACCESS_KEY_ID] = aws_credentials.access_key
os.environ[environment_vars.AWS_SECRET_ACCESS_KEY] = aws_credentials.secret_key
os.environ[environment_vars.AWS_SESSION_TOKEN] = aws_credentials.token

ismailsimsek commented 1 year ago

mateusz commented 1 year ago

Same issue here - running from Fargate, using google-auth 2.23.3. Thanks @tomwphillips for the workaround - works well.

Please note in my case the reported error was OSError: [Errno 22] Invalid Argument

youcandanch commented 4 months ago

@mmalecki that commit you have on your fork seems to do the trick for this -- any chance you could open a PR for that so we can try to get it into mainline?

mmalecki commented 4 months ago

Funny you should say this, I wrote it, then went on a weekly break without having had a chance to test this out, and was just getting back to it. Glad to hear it's working for you, I will open a PR.

mmalecki commented 4 months ago

@youcandanch now open as #1556

Gekko0114 commented 3 months ago

I've encountered the same issue here. I am using ECS and get a Transport error. Could you fix it? I think this issue was opened two years ago. I hope this issue will be fixed soon.

Getting metadata from plugin failed with error: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/meta-data/iam/security-credentials (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f99909b5fd0>: Failed to establish a new connection: [Errno 22] Invalid argument'))

googleapis / google-auth-library-python

Incorrect AWS metadata server path when running on Fargate #1099