search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
9.04k stars 1.87k forks source link

describe_cache_clusters() response dict is missing ARN #2489

Closed rvodden closed 3 years ago

rvodden commented 4 years ago

Describe the bug The Boto3 1.14.7 documentation claims that describe_cache_clusters() will return a dict which includes an ARN Key. This key, and therefore value, is missing.

Steps to reproduce Run the following script:

import boto3
boto3.set_stream_logger('')
elasticache_client = boto3.client('elasticache', region_name='eu-west-2')
clusters = elasticache_client.describe_cache_clusters()
print('CacheClusterId' in clusters['CacheClusters'][0])
print('ARN' in clusters['CacheClusters'][0])

Expected behavior The method should return a dict which includes the ARN as specified here:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/elasticache.html#ElastiCache.Client.describe_cache_clusters

Debug logs Full stack trace by adding boto3.set_stream_logger('') to your code.

2020-06-20 08:52:16,920 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2020-06-20 08:52:16,921 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2020-06-20 08:52:16,922 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2020-06-20 08:52:16,923 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2020-06-20 08:52:16,923 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2020-06-20 08:52:16,924 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2020-06-20 08:52:16,924 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2020-06-20 08:52:16,926 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2020-06-20 08:52:16,927 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2020-06-20 08:52:16,927 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2020-06-20 08:52:16,927 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2020-06-20 08:52:16,953 botocore.credentials [DEBUG] Looking for credentials via: env
2020-06-20 08:52:16,953 botocore.credentials [INFO] Found credentials in environment variables.
2020-06-20 08:52:16,954 botocore.loaders [DEBUG] Loading JSON file: /Users/richard.vodden/Documents/Source/Babylon/tagging-utils/.venv/lib/python3.7/site-packages/botocore/data/endpoints.json
2020-06-20 08:52:16,959 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x102ebbd40>
2020-06-20 08:52:16,976 botocore.loaders [DEBUG] Loading JSON file: /Users/richard.vodden/Documents/Source/Babylon/tagging-utils/.venv/lib/python3.7/site-packages/botocore/data/elasticache/2015-02-02/service-2.json
2020-06-20 08:52:16,981 botocore.hooks [DEBUG] Event creating-client-class.elasticache: calling handler <function add_generate_presigned_url at 0x102e8f9e0>
2020-06-20 08:52:17,030 botocore.endpoint [DEBUG] Setting elasticache timeout as (60, 60)
2020-06-20 08:52:17,031 botocore.loaders [DEBUG] Loading JSON file: /Users/richard.vodden/Documents/Source/Babylon/tagging-utils/.venv/lib/python3.7/site-packages/botocore/data/_retry.json
2020-06-20 08:52:17,032 botocore.client [DEBUG] Registering retry handlers for service: elasticache
2020-06-20 08:52:17,033 botocore.hooks [DEBUG] Event before-parameter-build.elasticache.DescribeCacheClusters: calling handler <function generate_idempotent_uuid at 0x102ef10e0>
2020-06-20 08:52:17,033 botocore.hooks [DEBUG] Event before-call.elasticache.DescribeCacheClusters: calling handler <function inject_api_version_header_if_needed at 0x102ef2950>
2020-06-20 08:52:17,033 botocore.endpoint [DEBUG] Making request for OperationModel(name=DescribeCacheClusters) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.14.7 Python/3.7.6 Darwin/19.4.0 Botocore/1.17.7'}, 'body': {'Action': 'DescribeCacheClusters', 'Version': '2015-02-02'}, 'url': 'https://elasticache.eu-west-2.amazonaws.com/', 'context': {'client_region': 'eu-west-2', 'client_config': <botocore.config.Config object at 0x1036000d0>, 'has_streaming_input': False, 'auth_type': None}}
2020-06-20 08:52:17,034 botocore.hooks [DEBUG] Event request-created.elasticache.DescribeCacheClusters: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x103600090>>
2020-06-20 08:52:17,034 botocore.hooks [DEBUG] Event choose-signer.elasticache.DescribeCacheClusters: calling handler <function set_operation_specific_signer at 0x102eed5f0>
2020-06-20 08:52:17,036 botocore.auth [DEBUG] Calculating signature using v4 auth.
2020-06-20 08:52:17,036 botocore.auth [DEBUG] CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:elasticache.eu-west-2.amazonaws.com
x-amz-date:20200620T075217Z
x-amz-security-token:<redacted>

content-type;host;x-amz-date;x-amz-security-token
7d5fa81bb12b569f001c103e15accd6359ef8ba8bfc9cae29aaaf140ec73265b
2020-06-20 08:52:17,036 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20200620T075217Z
20200620/eu-west-2/elasticache/aws4_request
58c568e2c62c0fb3e104675d8ae09d434e35cb5a860723a7a66dd4ef089df82c
2020-06-20 08:52:17,038 botocore.auth [DEBUG] Signature:
7a4e7fc4161ecd7af9ed65e72fc719edbf16771616d1d7d16e9e5c042ba96375
2020-06-20 08:52:17,039 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://elasticache.eu-west-2.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'Boto3/1.14.7 Python/3.7.6 Darwin/19.4.0 Botocore/1.17.7', 'X-Amz-Date': b'20200620T075217Z', 'X-Amz-Security-Token': <redacted>
Credential=ASIAV7ISZOPHR3MUBJVM/20200620/eu-west-2/elasticache/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=7a4e7fc4161ecd7af9ed65e72fc719edbf16771616d1d7d16e9e5c042ba96375', 'Content-Length': '47'}>
2020-06-20 08:52:17,041 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): elasticache.eu-west-2.amazonaws.com:443
2020-06-20 08:52:17,244 urllib3.connectionpool [DEBUG] https://elasticache.eu-west-2.amazonaws.com:443 "POST / HTTP/1.1" 200 8283
2020-06-20 08:52:17,256 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'd8280bb9-eb1c-4e11-8f21-84ea279658c5', 'Content-Type': 'text/xml', 'Content-Length': '8283', 'vary': 'accept-encoding', 'Date': 'Sat, 20 Jun 2020 07:52:16 GMT'}
2020-06-20 08:52:17,256 botocore.parsers [DEBUG] Response body:
b'<DescribeCacheClustersResponse xmlns="http://elasticache.amazonaws.com/doc/2015-02-02/">\n  <DescribeCacheClustersResult>\n    <CacheClusters>\n      <CacheCluster>\n        <CacheSubnetGroupName>elasticache-subnet-group-uk-sandbox-eu-west-2</CacheSubnetGroupName>\n        <NotificationConfiguration>\n          <TopicStatus>inactive</TopicStatus>\n          <TopicArn>arn:aws:sns:eu-west-2:410745664463:sandbox-eu-west-2-cache-monitoring</TopicArn>\n        </NotificationConfiguration>\n        <PreferredAvailabilityZone>eu-west-2b</PreferredAvailabilityZone>\n        <AtRestEncryptionEnabled>false</AtRestEncryptionEnabled>\n        <CacheClusterCreateTime>2020-01-22T13:40:57.240Z</CacheClusterCreateTime>\n        <SnapshotRetentionLimit>0</SnapshotRetentionLimit>\n        <CacheClusterStatus>available</CacheClusterStatus>\n        <NumCacheNodes>1</NumCacheNodes>\n        <Engine>redis</Engine>\n        <CacheSecurityGroups/>\n        <CacheParameterGroup>\n          <ParameterApplyStatus>in-sync</ParameterApplyStatus>\n          <CacheNodeIdsToReboot/>\n          <CacheParameterGroupName>default.redis5.0</CacheParameterGroupName>\n        </CacheParameterGroup>\n        <EngineVersion>5.0.6</EngineVersion>\n        <CacheClusterId>sandbox-uk-test-001</CacheClusterId>\n        <AuthTokenEnabled>false</AuthTokenEnabled>\n        <ClientDownloadLandingPage>https://console.aws.amazon.com/elasticache/home#client-download:</ClientDownloadLandingPage>\n        <SnapshotWindow>00:30-01:30</SnapshotWindow>\n        <CacheNodeType>cache.t2.micro</CacheNodeType>\n        <PendingModifiedValues/>\n        <PreferredMaintenanceWindow>wed:04:00-wed:05:00</PreferredMaintenanceWindow>\n        <SecurityGroups>\n          <member>\n            <SecurityGroupId>sg-0787db10275282f2b</SecurityGroupId>\n            <Status>active</Status>\n          </member>\n        </SecurityGroups>\n        <TransitEncryptionEnabled>false</TransitEncryptionEnabled>\n        <ReplicationGroupId>sandbox-uk-test</ReplicationGroupId>\n        <AutoMinorVersionUpgrade>true</AutoMinorVersionUpgrade>\n      </CacheCluster>\n      <CacheCluster>\n        <CacheSubnetGroupName>elasticache-subnet-group-uk-sandbox-eu-west-2</CacheSubnetGroupName>\n        <NotificationConfiguration>\n          <TopicStatus>inactive</TopicStatus>\n          <TopicArn>arn:aws:sns:eu-west-2:410745664463:sandbox-eu-west-2-cache-monitoring</TopicArn>\n        </NotificationConfiguration>\n        <PreferredAvailabilityZone>eu-west-2c</PreferredAvailabilityZone>\n        <AtRestEncryptionEnabled>false</AtRestEncryptionEnabled>\n        <CacheClusterCreateTime>2020-01-22T13:40:57.240Z</CacheClusterCreateTime>\n        <SnapshotRetentionLimit>14</SnapshotRetentionLimit>\n        <CacheClusterStatus>available</CacheClusterStatus>\n        <NumCacheNodes>1</NumCacheNodes>\n        <Engine>redis</Engine>\n        <CacheSecurityGroups/>\n        <CacheParameterGroup>\n          <ParameterApplyStatus>in-sync</ParameterApplyStatus>\n          <CacheNodeIdsToReboot/>\n          <CacheParameterGroupName>default.redis5.0</CacheParameterGroupName>\n        </CacheParameterGroup>\n        <EngineVersion>5.0.6</EngineVersion>\n        <CacheClusterId>sandbox-uk-test-002</CacheClusterId>\n        <AuthTokenEnabled>false</AuthTokenEnabled>\n        <ClientDownloadLandingPage>https://console.aws.amazon.com/elasticache/home#client-download:</ClientDownloadLandingPage>\n        <SnapshotWindow>00:30-01:30</SnapshotWindow>\n        <CacheNodeType>cache.t2.micro</CacheNodeType>\n        <PendingModifiedValues/>\n        <PreferredMaintenanceWindow>wed:04:00-wed:05:00</PreferredMaintenanceWindow>\n        <SecurityGroups>\n          <member>\n            <SecurityGroupId>sg-0787db10275282f2b</SecurityGroupId>\n            <Status>active</Status>\n          </member>\n        </SecurityGroups>\n        <TransitEncryptionEnabled>false</TransitEncryptionEnabled>\n        <ReplicationGroupId>sandbox-uk-test</ReplicationGroupId>\n        <AutoMinorVersionUpgrade>true</AutoMinorVersionUpgrade>\n      </CacheCluster>\n      <CacheCluster>\n        <AuthTokenLastModifiedDate>2020-02-26T04:11:43.808Z</AuthTokenLastModifiedDate>\n        <CacheSubnetGroupName>elasticache-subnet-group-uk-sandbox-eu-west-2</CacheSubnetGroupName>\n        <NotificationConfiguration>\n          <TopicStatus>inactive</TopicStatus>\n          <TopicArn>arn:aws:sns:eu-west-2:410745664463:sandbox-eu-west-2-cache-monitoring</TopicArn>\n        </NotificationConfiguration>\n        <PreferredAvailabilityZone>eu-west-2c</PreferredAvailabilityZone>\n        <AtRestEncryptionEnabled>true</AtRestEncryptionEnabled>\n        <CacheClusterCreateTime>2019-05-14T15:57:21.522Z</CacheClusterCreateTime>\n        <SnapshotRetentionLimit>0</SnapshotRetentionLimit>\n        <CacheClusterStatus>available</CacheClusterStatus>\n        <NumCacheNodes>1</NumCacheNodes>\n        <Engine>redis</Engine>\n        <CacheSecurityGroups/>\n        <CacheParameterGroup>\n          <ParameterApplyStatus>in-sync</ParameterApplyStatus>\n          <CacheNodeIdsToReboot/>\n          <CacheParameterGroupName>default.redis5.0</CacheParameterGroupName>\n        </CacheParameterGroup>\n        <EngineVersion>5.0.3</EngineVersion>\n        <CacheClusterId>sandbox-uk-test-tls-001</CacheClusterId>\n        <AuthTokenEnabled>true</AuthTokenEnabled>\n        <ClientDownloadLandingPage>https://console.aws.amazon.com/elasticache/home#client-download:</ClientDownloadLandingPage>\n        <SnapshotWindow>00:30-01:30</SnapshotWindow>\n        <CacheNodeType>cache.t2.micro</CacheNodeType>\n        <PendingModifiedValues/>\n        <PreferredMaintenanceWindow>wed:04:00-wed:05:00</PreferredMaintenanceWindow>\n        <SecurityGroups>\n          <member>\n            <SecurityGroupId>sg-0787db10275282f2b</SecurityGroupId>\n            <Status>active</Status>\n          </member>\n        </SecurityGroups>\n        <TransitEncryptionEnabled>true</TransitEncryptionEnabled>\n        <ReplicationGroupId>sandbox-uk-test-tls</ReplicationGroupId>\n        <AutoMinorVersionUpgrade>true</AutoMinorVersionUpgrade>\n      </CacheCluster>\n      <CacheCluster>\n        <AuthTokenLastModifiedDate>2020-04-09T00:34:40.772Z</AuthTokenLastModifiedDate>\n        <CacheSubnetGroupName>elasticache-subnet-group-uk-sandbox-eu-west-2</CacheSubnetGroupName>\n        <NotificationConfiguration>\n          <TopicStatus>inactive</TopicStatus>\n          <TopicArn>arn:aws:sns:eu-west-2:410745664463:sandbox-eu-west-2-cache-monitoring</TopicArn>\n        </NotificationConfiguration>\n        <PreferredAvailabilityZone>eu-west-2a</PreferredAvailabilityZone>\n        <AtRestEncryptionEnabled>true</AtRestEncryptionEnabled>\n        <CacheClusterCreateTime>2019-05-14T15:57:21.522Z</CacheClusterCreateTime>\n        <SnapshotRetentionLimit>14</SnapshotRetentionLimit>\n        <CacheClusterStatus>available</CacheClusterStatus>\n        <NumCacheNodes>1</NumCacheNodes>\n        <Engine>redis</Engine>\n        <CacheSecurityGroups/>\n        <CacheParameterGroup>\n          <ParameterApplyStatus>in-sync</ParameterApplyStatus>\n          <CacheNodeIdsToReboot/>\n          <CacheParameterGroupName>default.redis5.0</CacheParameterGroupName>\n        </CacheParameterGroup>\n        <EngineVersion>5.0.3</EngineVersion>\n        <CacheClusterId>sandbox-uk-test-tls-002</CacheClusterId>\n        <AuthTokenEnabled>true</AuthTokenEnabled>\n        <ClientDownloadLandingPage>https://console.aws.amazon.com/elasticache/home#client-download:</ClientDownloadLandingPage>\n        <SnapshotWindow>00:30-01:30</SnapshotWindow>\n        <CacheNodeType>cache.t2.micro</CacheNodeType>\n        <PendingModifiedValues/>\n        <PreferredMaintenanceWindow>wed:04:00-wed:05:00</PreferredMaintenanceWindow>\n        <SecurityGroups>\n          <member>\n            <SecurityGroupId>sg-0787db10275282f2b</SecurityGroupId>\n            <Status>active</Status>\n          </member>\n        </SecurityGroups>\n        <TransitEncryptionEnabled>true</TransitEncryptionEnabled>\n        <ReplicationGroupId>sandbox-uk-test-tls</ReplicationGroupId>\n        <AutoMinorVersionUpgrade>true</AutoMinorVersionUpgrade>\n      </CacheCluster>\n    </CacheClusters>\n  </DescribeCacheClustersResult>\n  <ResponseMetadata>\n    <RequestId>d8280bb9-eb1c-4e11-8f21-84ea279658c5</RequestId>\n  </ResponseMetadata>\n</DescribeCacheClustersResponse>\n'
2020-06-20 08:52:17,266 botocore.hooks [DEBUG] Event needs-retry.elasticache.DescribeCacheClusters: calling handler <botocore.retryhandler.RetryHandler object at 0x103600190>
2020-06-20 08:52:17,266 botocore.retryhandler [DEBUG] No retry needed.
True
False
swetashre commented 4 years ago

@rvodden - Thank you for your post. Boto3 documentation is generated from API reference https://docs.aws.amazon.com/AmazonElastiCache/latest/APIReference/API_DescribeCacheClusters.html and looks like it uses a general CacheCluster shape for the response. That's why the response includes all the attributes even if the api does not return that.

slushysnowman commented 4 years ago

Also experiencing same issue in 1.14.20

harisspa commented 4 years ago

I am using this feature on 5-Oct-2020 and still facing this issue. I am unable to get the ARN, which in turn will help me to fetch the tags. When we are getting the ARN from AWS CLI when we execute describe-cache-clusters why can't we get that in Boto3. The ease of using Boto3 relies on the exact matching with CLI.

swetashre commented 3 years ago

I am able to get ARN with boto3. Is anyone still facing the issue ?

In [8]: print('ARN' in response['CacheClusters'][0])
True
harisspa commented 3 years ago

@swetashre I can see the arn now. It wasn't available when I tried in Oct. Thanks for the update.

swetashre commented 3 years ago

I am closing this issue as this has been fixed. Please let us know if anyone has still any concerns.

github-actions[bot] commented 3 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

harisspa commented 3 years ago

Hi @swetashre and Others,

I don't see the ARN now. When we do a describe-cache-clusters through AWS CLI, I can see the ARN. But not through Boto3.

print('CacheClusterId' in clusters['CacheClusters'][0]) True print('ARN' in clusters['CacheClusters'][0]) False`