CredentialRetrievalError on ECS after a few successful calls

[sfc-gh-afedorov]

• Async AWS SDK for Python version: 9.3.1
• Python version: 3.7.11
• Operating System: ECS with Docker Container built on python:3.7-slim-stretch

Description

Trying to run aws_collect.py from the SnowAlert project

Everything works fine running on macOS with py3.9 but in ECS with credentials passed as environment variables, out of four times that I tried it in ECS Tasks, 0, 1, 8, and 12 calls are made before a novel exception is raised in connectors/utils.py:92

async with aioboto3.Session().client('sts') as sts:

botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from container-role: Error retrieving metadata: Received error when attempting to retrieve ECS metadata:

What I Did

Ran the connector on one of our internal accounts that works fine with 8.3.0 but raises an exception after 12 calls in v9

[terricain]

This would probably be an issue with aiobotocore, as aioboto3 uses its credential handling.

[sfc-gh-afedorov]

For posterity, similar newer issue has to do with the metadata API having what appears to be a rate limit that can be set via ECS_TASK_METADATA_RPS_LIMIT envar and documented here and apparent in response header X-Rate-Limit-Limit: 40.00. Newer issue showed up as —

Error when retrieving credentials from container-role: Error retrieving metadata: Received error when attempting to retrieve container metadata: Connect timeout on endpoint URL: "http://169.254.170.2/v2/credentials/..."