search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
9.04k stars 1.87k forks source link

AttributeError: 'str' object has no attribute 'get' #2350

Closed GreyArea765 closed 4 years ago

GreyArea765 commented 4 years ago

Just started to learn boto3 with NetApp StorageGRID, I'm receiving the above exception when trying to list buckets from "S3 Compatible" storage (NetApp StorageGRID). Please note that this only happens when the code is run from a certain network, it runs fine elsewhere so this is a bit of a head scratcher. I don't have a problem using tools such as Cloudberry explorer in this network using the same endpoint/credentials so I'd like to try and understand what is causing the exception.

See the below code,configure access key and secret key as needed. The ENDPOINT_URL is configured as

Config_Class

class Config(object):
    ACCESS_KEY = 'accesskey'
    SECRET_KEY = 'secretkey'
    ENDPOINT_URL = 'https://s3.somedomain.local'
    Config.REGION = 'us-east-1'

Test_Code

import boto3
from config import Config

ACCESS_KEY = Config.ACCESS_KEY
SECRET_KEY = Config.SECRET_KEY
ENDPOINT_URL = Config.ENDPOINT_URL
REGION = Config.REGION

boto3.set_stream_logger('')

session = boto3.session.Session(aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    region_name=REGION)

s3_client = session.client('s3', endpoint_url=ENDPOINT_URL)

response = s3_client.list_buckets()

# Output the bucket names
print('Existing buckets:')
for bucket in response['Buckets']:
    print('-- {}'.format(bucket['Name']))

Traceback

$ python3 s3_test.py 
2020-03-18 09:15:37,196 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2020-03-18 09:15:37,202 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2020-03-18 09:15:37,203 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2020-03-18 09:15:37,206 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2020-03-18 09:15:37,208 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2020-03-18 09:15:37,209 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2020-03-18 09:15:37,210 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2020-03-18 09:15:37,216 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2020-03-18 09:15:37,217 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2020-03-18 09:15:37,217 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2020-03-18 09:15:37,217 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2020-03-18 09:15:37,218 botocore.session [DEBUG] Setting config variable for region to 'us-east-1'
2020-03-18 09:15:37,225 botocore.loaders [DEBUG] Loading JSON file: /home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/data/endpoints.json
2020-03-18 09:15:37,249 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f27cfe29a70>
2020-03-18 09:15:37,807 botocore.loaders [DEBUG] Loading JSON file: /home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/data/s3/2006-03-01/service-2.json
2020-03-18 09:15:37,852 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f27cfe00320>
2020-03-18 09:15:37,852 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f27d0d25e60>
2020-03-18 09:15:37,931 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f27cfe000e0>
Traceback (most recent call last):
  File "s3_test.py", line 15, in <module>
    s3_client = session.client('s3', endpoint_url=ENDPOINT_URL)
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/session.py", line 835, in create_client
    client_config=config, api_version=api_version)
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/client.py", line 85, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/client.py", line 287, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/args.py", line 73, in get_client_args
    endpoint_url, is_secure, scoped_config)
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/args.py", line 153, in compute_client_args
    s3_config=s3_config,
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/args.py", line 215, in _compute_endpoint_config
    s3_config=s3_config, **resolve_endpoint_kwargs)
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/args.py", line 228, in _compute_s3_endpoint_config
    endpoint_config, resolve_endpoint_kwargs['endpoint_bridge'])
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/args.py", line 261, in _set_region_if_custom_s3_endpoint
    endpoint = endpoint_bridge.resolve('s3')
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/client.py", line 364, in resolve
    resolved, service_name, region_name, endpoint_url, is_secure)
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/client.py", line 382, in _create_endpoint
    if self._is_s3_dualstack_mode(service_name):
  File "/home/mattb/.local/share/virtualenvs/boto3_s3test-aHfp37jc/lib/python3.7/site-packages/botocore/client.py", line 414, in _is_s3_dualstack_mode
    enabled = self.scoped_config.get('s3', {}).get(
AttributeError: 'str' object has no attribute 'get'
swetashre commented 4 years ago

@GreyArea765 - Thank you for your post. As you have already mentioned that it is only reproducible in certain type of network, can you please provide me some details about the network where you are getting this error ? Have you configured credentials in that environment except providing in session ?

I only reproduced this issue when 's3' parameter is set as string type in ~/.aws/config file. My guess is that somehow your code is using credentials where 's3' parameter is set wrongly in config file. But i would need some more information about the environment in that network to reproduce the issue.

scoped_config.get('s3', {}) this value should be of type dict but it is giving string type. Can you please print the value of scoped_config before this line enabled = self.scoped_config.get('s3', {}).get( 'use_dualstack_endpoint', False) where you are getting error to see whether your scoped config contains 's3' parameter or not ?

no-response[bot] commented 4 years ago

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.