Connecting to SQS in docker after assume role/kubernetes IAM role not working

[eldarnegrinperion]

Please fill out the sections below to help us address your issue.

What issue did you see ? logs-from-kubernetes.txt when inside docker, can't access role assumed on computer/iam role on kubernetes from my computer it works fine, it finds the credential and config files. when creating s3 client all works fine. this happens only in sqs client..

Steps to reproduce If you have a runnable example, please include it as a snippet or link to a repository/gist for larger code examples. simple python (3.7.4) code, boto3 (1.14.2), just creating a client for sqs. if __name__ == '__main__': boto3.set_stream_logger('') sqs = boto3.client('sqs')

Debug logs Full stack trace by adding boto3.set_stream_logger('') to your code. here is local docker, and attached kubernetes logs file

2020-07-02 07:05:24,593 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2020-07-02 07:05:24,597 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2020-07-02 07:05:24,598 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2020-07-02 07:05:24,602 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2020-07-02 07:05:24,602 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2020-07-02 07:05:24,604 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2020-07-02 07:05:24,605 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2020-07-02 07:05:24,612 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2020-07-02 07:05:24,613 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2020-07-02 07:05:24,613 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2020-07-02 07:05:24,613 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: env
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: sso
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: config-file
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2020-07-02 07:05:24,634 botocore.credentials [DEBUG] Looking for credentials via: container-role
2020-07-02 07:05:24,634 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2020-07-02 07:05:24,635 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 169.254.169.254:80
2020-07-02 07:05:25,646 urllib3.connectionpool [DEBUG] Starting new HTTP connection (2): 169.254.169.254:80
2020-07-02 07:05:26,660 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1336, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 379, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 428, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: AWSHTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 342, in _get_request
    response = self._session.send(request.prepare())
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2020-07-02 07:05:26,669 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2020-07-02 07:05:26,671 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json
2020-07-02 07:05:26,681 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f503ec53b00>
2020-07-02 07:05:26,696 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/sqs/2012-11-05/service-2.json
2020-07-02 07:05:26,701 botocore.hooks [DEBUG] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f503eca0f80>
Traceback (most recent call last):
  File "EnrichmentWorkerService.py", line 88, in <module>
    sqs = boto3.client('sqs')
  File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 91, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 835, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 85, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 287, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 73, in get_client_args
    endpoint_url, is_secure, scoped_config)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 153, in compute_client_args
    s3_config=s3_config,
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 218, in _compute_endpoint_config
    return self._resolve_endpoint(**resolve_endpoint_kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 301, in _resolve_endpoint
    service_name, region_name, endpoint_url, is_secure)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 361, in resolve
    service_name, region_name)
  File "/usr/local/lib/python3.7/site-packages/botocore/regions.py", line 134, in construct_endpoint
    partition, service_name, region_name)
  File "/usr/local/lib/python3.7/site-packages/botocore/regions.py", line 148, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

[swetashre]

@eldarnegrinperion - Thank you for your post. It looks like this issue might be related to your network policies for the particular container. https://kubernetes.io/docs/concepts/services-networking/network-policies/ Do you have any restrictive security group for the particular container because of which it is not able to connect to IMDS ? How have you configured your container to use assume role ?

This stack overflow post might help in debugging the issue: https://stackoverflow.com/questions/22409367/fetching-aws-instance-metadata-from-within-docker-container

[eldarnegrinperion]

no special configuration. it happens in my local environment as well - when i run the code directly (virtualenv) all is good. when i run it in a docker (very simple python docker, no special rules) it doesn't. when i change the code to S3 client, all is good and working as expected why does it happen when trying to create a sqs client?

[eldarnegrinperion]

Dockerfile.txt

[swetashre]

@eldarnegrinperion - Thanks for responding. I am not able to reproduce the issue with the Dockerfile you provided. I am assuming you are running this docker container in an ec2 instance as ec2 instance metadata can't be used from your local environment. Are you able to use any api call with the s3 client ? What is the response you are getting when using curl http://169.254.169.254/latest/meta-data/iam/security-credentials from your docker container ?

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[eldarnegrinperion]

sorry i missed your response. i can't even load the docker as it fails and exits as the log above i sent. i will try to run the docker with just an endless loop so i can get in exec mode and do the CURL and update results here. thanks

[eldarnegrinperion]

this is what i get when exec CURL to the above mentioned url from within the docker on my machine.

curl: (56) Recv failure: Connection reset by peer

[eldarnegrinperion]

and here's my code, pretty simple: ` if name == 'main':

boto3.set_stream_logger('')

print('starting polling...')
Path('/tmp/healty').touch()  # health check for probe

while True:
    sleep_timer=random() * 5 * 60
    print(f'{datetime.utcnow()} loop {sleep_timer} seconds...')
    time.sleep(sleep_timer)

# sqs_polling(queue_url=settings.SQS_URL,
#             callback=sqs_message_callback,
#             max_workers=settings.MAX_WORKERS,
#             process_worker=settings.PROCESS_WORKER,
#             interval_seconds=settings.INTERVAL_SECONDS
#             )

`

[swetashre]

Sorry for late reply. Here the problem is that you are trying to use an assume role from the docker container but when boto3 is not able to find that role it is trying to use ec2 instance metadata service and then it is giving you error.

Have you tried by adding the credential file to your docker container root folder and see if that works for you ? Can you please follow these steps and let me know your result?

1. docker images
2. docker run -it -d <your image id>
3. docker ps # you will get container id for you image
4. docker attach <container id>

after running this command you will get something like this:
 root@a1bf5930c4ce:/# 

then execute this code
root@a1bf5930c4ce:/# python
Python 2.7.17 (default, Apr 15 2020, 17:20:14) 
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>>boto3.set_stream_logger('')
>>> client = boto3.client('sqs')

If you are getting the same credentials error can you add your ~/.aws folder to the root folder and again run these command ?

[eldarnegrinperion]

I will check and get back here.

One note though: I don't understand why for S3 it works ok and for SQS not? If the AWS session is expired S3 fails as well, then I assume role and all works fine..

[eldarnegrinperion]

still no good. root@7a7c1bcb49b7:/usr/src/proj# ls /.aws config credentials root@7a7c1bcb49b7:/usr/src/proj# python Python 3.7.4 (default, Oct 17 2019, 05:59:21) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import boto3 boto3.set_stream_logger('') client = boto3.client('sqs') 2020-07-23 15:17:37,467 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2020-07-23 15:17:37,470 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway 2020-07-23 15:17:37,472 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2020-07-23 15:17:37,476 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2020-07-23 15:17:37,477 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2020-07-23 15:17:37,478 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2020-07-23 15:17:37,479 botocore.hooks [DEBUG] Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2020-07-23 15:17:37,486 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2020-07-23 15:17:37,486 botocore.hooks [DEBUG] Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2020-07-23 15:17:37,486 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2020-07-23 15:17:37,486 botocore.hooks [DEBUG] Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: env 2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: assume-role 2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity 2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: sso 2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file 2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: custom-process 2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: config-file 2020-07-23 15:17:37,495 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file 2020-07-23 15:17:37,495 botocore.credentials [DEBUG] Looking for credentials via: boto-config 2020-07-23 15:17:37,495 botocore.credentials [DEBUG] Looking for credentials via: container-role 2020-07-23 15:17:37,495 botocore.credentials [DEBUG] Looking for credentials via: iam-role 2020-07-23 15:17:37,496 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 169.254.169.254:80 2020-07-23 15:17:38,510 urllib3.connectionpool [DEBUG] Starting new HTTP connection (2): 169.254.169.254:80 2020-07-23 15:17:39,524 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/" Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/usr/local/lib/python3.7/http/client.py", line 1336, in getresponse response.begin() File "/usr/local/lib/python3.7/http/client.py", line 306, in begin version, status, reason = self._read_status() File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/local/lib/python3.7/socket.py", line 589, in readinto return self._sock.recv_into(b) socket.timeout: timed out

[swetashre]

@eldarnegrinperion - Are you running this docker inside a ec2 instance or from your local environment ?

[eldarnegrinperion]

i tried two configurations:

1. local docker on my machine, with credential files (temp session)
2. inside kubernetes cluster in AWS, which has access to anything inside the account through role (and this is tested in other containers in different languages) both yield the same result.

[swetashre]

When you are running in local environment did you add your credentials file to your docker container root folder ? Are you also getting the same error even after adding the credentials file to your docker container root folder ?

[eldarnegrinperion]

yes i am bringing it in, /.aws/credentials + /.aws/config, and still get the error my credentials file holds: [default] aws_access_key_id = xxx aws_secret_access_key = xxx region = us-east-1 aws_session_token = xxx

my config file holds: [default] output = json region = us-east-1

[swetashre]

Are you getting the same error even after bringing ~/.aws/credentials and ~/.aws/config file to the root folder of your docker container ?

You should not get that same error if your credentials file is present in the root folder of docker container. Can you provide me debug logs ?

[eldarnegrinperion]

on local machine:

2020-08-31 17:51:44,005 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2020-08-31 17:51:44,008 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway 2020-08-31 17:51:44,010 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2020-08-31 17:51:44,014 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2020-08-31 17:51:44,014 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2020-08-31 17:51:44,015 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2020-08-31 17:51:44,017 botocore.hooks [DEBUG] Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2020-08-31 17:51:44,021 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2020-08-31 17:51:44,021 botocore.hooks [DEBUG] Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2020-08-31 17:51:44,021 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2020-08-31 17:51:44,021 botocore.hooks [DEBUG] Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2020-08-31 17:51:44,036 botocore.credentials [DEBUG] Looking for credentials via: env 2020-08-31 17:51:44,036 botocore.credentials [DEBUG] Looking for credentials via: assume-role 2020-08-31 17:51:44,036 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity 2020-08-31 17:51:44,036 botocore.credentials [DEBUG] Looking for credentials via: sso 2020-08-31 17:51:44,037 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file 2020-08-31 17:51:44,037 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials 2020-08-31 17:51:44,041 botocore.loaders [DEBUG] Loading JSON file: /usr/local/src/ca-sbp-events-enrichment/venv/lib/python3.7/site-packages/botocore/data/endpoints.json 2020-08-31 17:51:44,082 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f3580f6e050> 2020-08-31 17:51:44,100 botocore.loaders [DEBUG] Loading JSON file: /usr/local/src/ca-sbp-events-enrichment/venv/lib/python3.7/site-packages/botocore/data/sqs/2012-11-05/service-2.json 2020-08-31 17:51:44,105 botocore.hooks [DEBUG] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f3580fbdb90> 2020-08-31 17:51:44,110 botocore.endpoint [DEBUG] Setting sqs timeout as (60, 60) 2020-08-31 17:51:44,111 botocore.loaders [DEBUG] Loading JSON file: /usr/local/src/ca-sbp-events-enrichment/venv/lib/python3.7/site-packages/botocore/data/_retry.json 2020-08-31 17:51:44,112 botocore.client [DEBUG] Registering retry handlers for service: sqs

[eldarnegrinperion]

inside docker:

2020-08-31 14:56:34,745 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2020-08-31 14:56:34,749 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway 2020-08-31 14:56:34,750 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2020-08-31 14:56:34,754 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2020-08-31 14:56:34,755 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2020-08-31 14:56:34,756 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2020-08-31 14:56:34,758 botocore.hooks [DEBUG] Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2020-08-31 14:56:34,764 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2020-08-31 14:56:34,765 botocore.hooks [DEBUG] Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2020-08-31 14:56:34,765 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2020-08-31 14:56:34,765 botocore.hooks [DEBUG] Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2020-08-31 14:56:34,797 botocore.credentials [DEBUG] Looking for credentials via: env 2020-08-31 14:56:34,797 botocore.credentials [DEBUG] Looking for credentials via: assume-role 2020-08-31 14:56:34,798 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity 2020-08-31 14:56:34,798 botocore.credentials [DEBUG] Looking for credentials via: sso 2020-08-31 14:56:34,798 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file 2020-08-31 14:56:34,798 botocore.credentials [DEBUG] Looking for credentials via: custom-process 2020-08-31 14:56:34,798 botocore.credentials [DEBUG] Looking for credentials via: config-file 2020-08-31 14:56:34,798 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file 2020-08-31 14:56:34,799 botocore.credentials [DEBUG] Looking for credentials via: boto-config 2020-08-31 14:56:34,799 botocore.credentials [DEBUG] Looking for credentials via: container-role 2020-08-31 14:56:34,799 botocore.credentials [DEBUG] Looking for credentials via: iam-role 2020-08-31 14:56:34,800 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 169.254.169.254:80 2020-08-31 14:56:35,812 urllib3.connectionpool [DEBUG] Starting new HTTP connection (2): 169.254.169.254:80 2020-08-31 14:56:36,828 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/" Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/usr/local/lib/python3.7/http/client.py", line 1336, in getresponse response.begin() File "/usr/local/lib/python3.7/http/client.py", line 306, in begin version, status, reason = self._read_status() File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/local/lib/python3.7/socket.py", line 589, in readinto return self._sock.recv_into(b) socket.timeout: timed out

[eldarnegrinperion]

all same code, just inside and outside docker.

[swetashre]

On your local machine logs i am not seeing any error and i can see that botocore is successfully able to find the credentials.

But inside docker botocore is not able to find the credentials in the shared credential file but if you have added your credentials file to your docker container root folder then it should be able to find the credentials. Please make sure you have credentials present in the docker container root folder.

These links contain some useful information: https://stackoverflow.com/questions/36354423/which-is-the-best-way-to-pass-aws-credentials-to-docker-container https://aws.amazon.com/premiumsupport/knowledge-center/codebuild-temporary-credentials-docker/

[eldarnegrinperion]

i made sure of it.. we talked about it several times now. i exec'ed into the pod and verified the credentials are there, were copied from the computer. still, you see the logs. that's exactly why i opened the issue in the first place.

[kdaily]

HI @eldarnegrinperion, I've reviewed this issue. Are you still experiencing trouble with this, potentially after updating your boto3 version? I've used Docker with boto3 successfully to use both an s3 and SQS client, so I'm not able to reproduce either.

[eldarnegrinperion]

hi, i've updated boto3 to 1.17.59 and still this issue.

2021-04-28 07:51:11,233 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2021-04-28 07:51:11,239 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2021-04-28 07:51:11,241 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2021-04-28 07:51:11,250 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2021-04-28 07:51:11,250 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2021-04-28 07:51:11,252 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2021-04-28 07:51:11,254 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2021-04-28 07:51:11,266 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2021-04-28 07:51:11,266 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2021-04-28 07:51:11,266 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2021-04-28 07:51:11,267 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2021-04-28 07:51:11,276 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2021-04-28 07:51:11,282 botocore.credentials [DEBUG] Looking for credentials via: env
2021-04-28 07:51:11,283 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2021-04-28 07:51:11,283 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2021-04-28 07:51:11,283 botocore.credentials [DEBUG] Looking for credentials via: sso
2021-04-28 07:51:11,283 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2021-04-28 07:51:11,283 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2021-04-28 07:51:11,283 botocore.credentials [DEBUG] Looking for credentials via: config-file
2021-04-28 07:51:11,284 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2021-04-28 07:51:11,284 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2021-04-28 07:51:11,284 botocore.credentials [DEBUG] Looking for credentials via: container-role
2021-04-28 07:51:11,284 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2021-04-28 07:51:11,285 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 169.254.169.254:80
2021-04-28 07:51:12,287 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 323, in send
    chunked=self._chunked(request.headers),
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 379, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/site-packages/botocore/awsrequest.py", line 93, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/site-packages/botocore/awsrequest.py", line 120, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/site-packages/botocore/awsrequest.py", line 204, in send
    return super(AWSConnection, self).send(str)
  File "/usr/local/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7f000c8e3850>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 377, in _fetch_metadata_token
    response = self._session.send(request.prepare())
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 347, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
2021-04-28 07:51:12,293 urllib3.connectionpool [DEBUG] Starting new HTTP connection (2): 169.254.169.254:80
2021-04-28 07:51:13,295 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 323, in send
    chunked=self._chunked(request.headers),
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 379, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/site-packages/botocore/awsrequest.py", line 93, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/site-packages/botocore/awsrequest.py", line 120, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/site-packages/botocore/awsrequest.py", line 204, in send
    return super(AWSConnection, self).send(str)
  File "/usr/local/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7f000cc93dd0>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 425, in _get_request
    response = self._session.send(request.prepare())
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 347, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2021-04-28 07:51:13,295 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2021-04-28 07:51:13,296 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json
2021-04-28 07:51:13,318 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f000d3c1ef0>
2021-04-28 07:51:13,340 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/sqs/2012-11-05/service-2.json
2021-04-28 07:51:13,346 botocore.hooks [DEBUG] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f000d3f79e0>
Traceback (most recent call last):
  File "/opt/project/EnrichmentWorkerService.py", line 91, in <module>
    sqs = boto3.client('sqs')
  File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 93, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 851, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 87, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 328, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 73, in get_client_args
    endpoint_url, is_secure, scoped_config)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 154, in compute_client_args
    s3_config=s3_config,
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 220, in _compute_endpoint_config
    return self._resolve_endpoint(**resolve_endpoint_kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 303, in _resolve_endpoint
    service_name, region_name, endpoint_url, is_secure)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 402, in resolve
    service_name, region_name)
  File "/usr/local/lib/python3.7/site-packages/botocore/regions.py", line 134, in construct_endpoint
    partition, service_name, region_name)
  File "/usr/local/lib/python3.7/site-packages/botocore/regions.py", line 148, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

Process finished with exit code 1

[eldarnegrinperion]

locally on my machine (without docker):

2021-04-28 10:55:27,470 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2021-04-28 10:55:27,475 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2021-04-28 10:55:27,478 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2021-04-28 10:55:27,483 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2021-04-28 10:55:27,483 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2021-04-28 10:55:27,484 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2021-04-28 10:55:27,486 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2021-04-28 10:55:27,527 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2021-04-28 10:55:27,528 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2021-04-28 10:55:27,528 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2021-04-28 10:55:27,528 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2021-04-28 10:55:27,579 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2021-04-28 10:55:27,585 botocore.credentials [DEBUG] Looking for credentials via: env
2021-04-28 10:55:27,586 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2021-04-28 10:55:27,586 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2021-04-28 10:55:27,586 botocore.credentials [DEBUG] Looking for credentials via: sso
2021-04-28 10:55:27,586 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2021-04-28 10:55:27,587 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2021-04-28 10:55:27,588 botocore.loaders [DEBUG] Loading JSON file: /usr/local/src/ca-sbp-events-enrichment/venv/lib/python3.7/site-packages/botocore/data/endpoints.json
2021-04-28 10:55:27,616 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fb9cab53a70>
2021-04-28 10:55:27,655 botocore.loaders [DEBUG] Loading JSON file: /usr/local/src/ca-sbp-events-enrichment/venv/lib/python3.7/site-packages/botocore/data/sqs/2012-11-05/service-2.json
2021-04-28 10:55:27,665 botocore.hooks [DEBUG] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7fb9cab810e0>
2021-04-28 10:55:27,686 botocore.endpoint [DEBUG] Setting sqs timeout as (60, 60)
2021-04-28 10:55:27,692 botocore.loaders [DEBUG] Loading JSON file: /usr/local/src/ca-sbp-events-enrichment/venv/lib/python3.7/site-packages/botocore/data/_retry.json
2021-04-28 10:55:27,693 botocore.client [DEBUG] Registering retry handlers for service: sqs

Process finished with exit code 0

[stobrien89]

Hi @eldarnegrinperion,

If you're still experiencing this issue, I'd be curious to see what the logs from your successful s3 call (from docker) look like. I suspect this may actually be a no region error, like the error message suggests, as botocore automatically handles s3 endpoint construction/region redirection if a region is not provided to a client. Can you try passing a region_name when you instantiate your sqs client?

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[eldarnegrinperion]

i passed a region_name and it worked. but it's still a bit weird and inconsistent. if i am stating a default region in the credentials file, it should be working, same as your example, same as i am doing from my local machine and works just fine, and same as other services (s3 for example)

[stobrien89]

Hi @eldarnegrinperion,

Thanks for letting me know! The default region should be specified in the config file, but that's also in the .aws directory. In Docker, I think this should go in /root/.aws/config instead of /.aws/... like you had noted before. Let me know if that works!

[eldarnegrinperion]

actually seems like the location of the .aws folder is different per base image.. if we're talking airflow for example, the location i got working is /home/airflow/.aws anyway, ran it without config, with config file, all the same, notice that the exception is the same regardless if i put region_name or not. WITH region_name:

boto3.client("sqs", region_name="eu-west-1")
2021-08-24 15:55:50,701 botocore.credentials [DEBUG] Looking for credentials via: env
2021-08-24 15:55:50,702 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2021-08-24 15:55:50,702 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2021-08-24 15:55:50,702 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2021-08-24 15:55:50,702 botocore.credentials [DEBUG] Looking for credentials via: config-file
2021-08-24 15:55:50,702 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2021-08-24 15:55:50,702 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2021-08-24 15:55:50,703 botocore.credentials [DEBUG] Looking for credentials via: container-role
2021-08-24 15:55:50,703 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2021-08-24 15:55:50,703 urllib3.util.retry [DEBUG] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2021-08-24 15:55:50,703 urllib3.connectionpool [DEBUG] Starting new HTTP connection (3): 169.254.169.254:80
2021-08-24 15:55:51,709 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 258, in send
    decode_content=False,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 386, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "/usr/local/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7fbf3190ae80>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/utils.py", line 303, in _get_request
    response = self._session.send(request.prepare())
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 282, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2021-08-24 15:55:51,710 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2021-08-24 15:55:51,711 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fbf31f01d08>
2021-08-24 15:55:51,711 botocore.hooks [DEBUG] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7fbf31f3b400>
2021-08-24 15:55:51,712 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
2021-08-24 15:55:51,714 botocore.endpoint [DEBUG] Setting sqs timeout as (60, 60)
2021-08-24 15:55:51,715 botocore.client [DEBUG] Registering retry handlers for service: sqs
<botocore.client.SQS object at 0x7fbf3190aeb8>

without region_name:

boto3.client("sqs")
2021-08-24 15:59:12,782 botocore.credentials [DEBUG] Looking for credentials via: env
2021-08-24 15:59:12,782 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2021-08-24 15:59:12,782 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2021-08-24 15:59:12,782 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2021-08-24 15:59:12,782 botocore.credentials [DEBUG] Looking for credentials via: config-file
2021-08-24 15:59:12,783 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2021-08-24 15:59:12,783 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2021-08-24 15:59:12,783 botocore.credentials [DEBUG] Looking for credentials via: container-role
2021-08-24 15:59:12,783 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2021-08-24 15:59:12,784 urllib3.util.retry [DEBUG] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2021-08-24 15:59:12,784 urllib3.connectionpool [DEBUG] Starting new HTTP connection (4): 169.254.169.254:80
2021-08-24 15:59:13,786 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 258, in send
    decode_content=False,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 386, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "/usr/local/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7fbf3190a780>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/utils.py", line 303, in _get_request
    response = self._session.send(request.prepare())
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 282, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2021-08-24 15:59:13,787 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2021-08-24 15:59:13,787 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fbf31f01d08>
2021-08-24 15:59:13,788 botocore.hooks [DEBUG] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7fbf31f3b400>
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/boto3/__init__.py", line 93, in client
    return _get_default_session().client(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/session.py", line 838, in create_client
    client_config=config, api_version=api_version)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/client.py", line 86, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/client.py", line 328, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/args.py", line 47, in get_client_args
    endpoint_url, is_secure, scoped_config)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/args.py", line 117, in compute_client_args
    service_name, region_name, endpoint_url, is_secure)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/client.py", line 402, in resolve
    service_name, region_name)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/regions.py", line 122, in construct_endpoint
    partition, service_name, region_name)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

and just for the sake of comparison, S3 client without region_name:

boto3.client("s3")
2021-08-24 15:59:38,249 botocore.credentials [DEBUG] Looking for credentials via: env
2021-08-24 15:59:38,249 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2021-08-24 15:59:38,250 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2021-08-24 15:59:38,250 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2021-08-24 15:59:38,250 botocore.credentials [DEBUG] Looking for credentials via: config-file
2021-08-24 15:59:38,250 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2021-08-24 15:59:38,250 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2021-08-24 15:59:38,250 botocore.credentials [DEBUG] Looking for credentials via: container-role
2021-08-24 15:59:38,251 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2021-08-24 15:59:38,251 urllib3.util.retry [DEBUG] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2021-08-24 15:59:38,251 urllib3.connectionpool [DEBUG] Starting new HTTP connection (5): 169.254.169.254:80
2021-08-24 15:59:39,256 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 258, in send
    decode_content=False,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 386, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "/usr/local/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7fbf31985e48>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/utils.py", line 303, in _get_request
    response = self._session.send(request.prepare())
  File "/home/airflow/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 282, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2021-08-24 15:59:39,256 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2021-08-24 15:59:39,257 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fbf31f01d08>
2021-08-24 15:59:39,259 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fbf31f3b620>
2021-08-24 15:59:39,259 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7fbf31e34a60>
2021-08-24 15:59:39,259 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fbf31f3b400>
2021-08-24 15:59:39,259 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
2021-08-24 15:59:39,261 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2021-08-24 15:59:39,263 botocore.client [DEBUG] Registering retry handlers for service: s3
2021-08-24 15:59:39,263 botocore.client [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
<botocore.client.S3 object at 0x7fbf31951f98>

[stobrien89]

@eldarnegrinperion,

Thanks for the update. Just to confirm, are you including the credentials file here as well? Or are you using the IAM role mentioned earlier? Thanks!

[eldarnegrinperion]

Credentials folder with credentials file and config file included in the docker container

[stobrien89]

Hi @eldarnegrinperion,

Thanks for confirming. Just to double-check, by credentials folder, you mean the .aws directory? If so, where in the docker container were they being included? Thanks for your patience.

[eldarnegrinperion]

hi @stobrien89, yes the .aws folder in ubuntu base image i put it in /home/ubuntu/.aws in airflow base image i put it in /home/airflow/.aws

[stobrien89]

Hi @eldarnegrinperion,

Thanks for your patience. I'm not entirely sure about airflow, but can you try moving the .aws folder in the ubuntu base image to home or whatever the equivalent of ~/.aws would be? Typically credentials will be read at ~/.aws/credentials and the config at ~/.aws/config.

[eldarnegrinperion]

as i said, in an ubuntu image, i put it at /home/ubuntu/.aws (equal to ~/.aws) when i put it at /.aws even S3 client didn't work

[stobrien89]

Thanks for the follow-up, @eldarnegrinperion. It looks like the location varies depending on the user as well. Are you logged in as the root user in your container? If so, the location will be:

root@ip-someip:~/.aws# pwd
/root/.aws

If not, it will be /home/ubuntu/.aws, like you mentioned.

I know the initial issue was related to the IAM role not working, but at this point, you just need to use the config file to declare your default region, correct?

[eldarnegrinperion]

correct.

[stobrien89]

For Ubuntu, did either of the paths work based on the user privileges you had in your container (root vs admin)? Like I said before, I'm not entirely sure about airflow, but their documentation seems to indicate the .aws directory can be /home/.aws as well. I'm sure this may also depend on the user/user privileges available.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[eldarnegrinperion]

if the .aws folder wasn't in the correct folder (i.e /.aws | /home/ubuntu/.aws | /home/airflow/.aws depending on image) nothing works - S3 and all other services as well. the issue here is even if the folder is in the correct place (verified by logging into S3 and performing actions) the default region from config file isn't being taken and hence causing the exception when trying to work with region specific services such as SQS.

[stobrien89]

Hi @eldarnegrinperion,

the issue here is even if the folder is in the correct place (verified by logging into S3 and performing actions) the default region from config file isn't being taken and hence causing the exception when trying to work with region specific services such as SQS.

In this case, were you still using an instance profile for credentials? or were you using the credentials file? Would you be able to share what your config file looks like?

[eldarnegrinperion]

yes i am bringing it in, /.aws/credentials + /.aws/config, and still get the error my credentials file holds: [default] aws_access_key_id = xxx aws_secret_access_key = xxx region = us-east-1 aws_session_token = xxx

my config file holds: [default] output = json region = us-east-1

it's here above :)

[stobrien89]

Hi @eldarnegrinperion,

Sorry for missing that. It's quite a long thread 😄. The only thing I see that's out of the ordinary is declaring a region in the credentials file (usually only specified in the config file), but I've tested it and that seems to work fine. As an alternative, have you tried setting the AWS_DEFAULT_REGION environment variable when using your instance profile?

[eldarnegrinperion]

with AWS default region it works fine. again, the issue here is that it's not consistent. it SHOULD support the configuration file, as stated in the docs, and it doesn't, which is what caused the confusion that started this thread :)

[tim-finnigan]

Hi @eldarnegrinperion, this issue was recently assigned to me so I wanted to check in. Is this still an issue, and if so do you have any updates as far as what you’ve tried doing?

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[eldarnegrinperion]

it's such an old issue, but still exists. eventually it is up to you guys to decide if you want to solve it. thanks for your time :)

[kdaily]

Hi @eldarnegrinperion,

I took your Dockerfile and can reproduce your error. I've determined the error is indeed caused by missing AWS configuration and credentials - they are not in the right place. The location of the AWS config and credentials file will change depending on the Docker container image you're using, so they'll need to be moved to the home directory of the user who is running the command. When using the base container image python:3.7.4, the container is run as root, and the home directory is /root/. This means your config and credentials files must be in /root/.aws/config and /root/.aws/credentials. I verified this by adding in the following to the Dockerfile, assuming your config and credential files are present in the current local directory (CAUTION: I would NOT recommend this method going forward, only to illustrate that this is the correct location):

## Add AWS config and credentials
RUN mkdir /root/.aws/
COPY config /root/.aws/
COPY credentials /root/.aws/

If I do not copy them, or copy them to /.aws/, I receive the same error as you, NoRegionError. I could also provide them by mounting the directory containing the config and credential files when running docker run via the -v flag.

There was some confusion as to why you seemed to get different behavior depending on which service client you were using - S3 or SQS. You can instantiate an S3 client without a region because it assumes you would be using the global region, so you will not receive a NoRegionError. However, you would eventually get a NoCredentialsError if you tried to run an S3 client command if your configuration file is missing (or, you haven't set a region using another method). Other clients, like SQS, require a region when constructing the client; if it cannot find a region using any of the methods available (IMDS, environment variable, parameter, or config file) you'll receive the NoRegionError.

I've ruled out any reason why boto3 would not be working, and this is an issue with your specific environment. We aren't able to assist on the details related to Kubernetes - you would be better off asking in a forum like repost.aws or Stack Overflow.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.