aws resourcegroupstaggingapi get-resources returns deleted resources

[j-land]

Describe the bug

I see discrepancies between the reported resources from the resourcegroupstaggingapi get-resources API and resources that actually exist either from individual service APIs or the console. The following resources seem to be impacted (though not sure if this is an exhaustive list):

• EC2 fleet
• EC2 volumes
• Route53 hosted zones
• Elastic Load Balancing listeners
• Elastic Load Balancing listener rules

I see these discrepancies when invoking the resourcegroupstaggingapi from both the AWS CLI and the boto3 library. Through manual inspection, I have found that the resourcegroupstaggingapi returns information for deleted resources.

I have found a number of related issues/forums/posts that describe the same issue but have largely been ignored or dismissed:

• https://github.com/aws/aws-cli/issues/4173
• https://repost.aws/questions/QUcinjyU36SBSmcYA8eYJf0Q/only-get-existing-resources-with-resourcegrouptaggingapi
• https://stackoverflow.com/questions/63876995/aws-resourcegroupstaggingapi-retrieving-resource-that-doesnt-exist
• https://stackoverflow.com/questions/71035336/only-get-existing-resources-with-resourcegrouptaggingapi

Expected Behavior

The resourcegroupstaggingapi returns information only for active resources OR provides an argument to filter deleted resources.

Current Behavior

The resourcegroupstaggingapi returns information for active and deleted resources.

Reproduction Steps

Route53

aws resourcegroupstaggingapi get-resources --resource-type-filters route53 | jq '.ResourceTagMappingList | map(.ResourceARN) | length'
70

aws route53 get-hosted-zone-count | jq '.HostedZoneCount'
21

EC2 Fleets

aws resourcegroupstaggingapi get-resources --resource-type-filters ec2:fleet | jq '.ResourceTagMappingList | map(.ResourceARN) | length'
4884

aws ec2 describe-fleets | jq '.Fleets | map(.FleetId) | length'
0

EC2 Volumes

aws resourcegroupstaggingapi get-resources --resource-type-filters ec2:volume | jq '.ResourceTagMappingList | map(.ResourceARN) | length'
15

aws ec2 describe-volumes | jq '.Volumes | length'
9

Possible Solution

No response

Additional Information/Context

No response

CLI version used

aws-cli/2.11.23

Environment details (OS name and version, etc.)

Darwin/22.6.0

[RyanFitzSimmonsAK]

Hi @j-land, thanks for raising this issue. I was able to reproduce this for some, but not all, of the resources you mentioned. Could you upgrade to the most recent version of AWS CLI, and let me know which of these resources still have this behavior?

[j-land]

Sure! I just upgraded to aws-cli/2.13.21 and reran the commands above.

Still seeing discrepancies with the following resources:

• EC2 fleet
• EC2 volumes
• Route53 hosted zones
• Elastic Load Balancing listeners
• Elastic Load Balancing listener rules

Additionally, I now see discrepancies with the following resources:

• EC2 instances
• EC2 network interfaces

For all of these resource types (except for ELB listeners/rules), the specific CLI client API (e.g., aws ec2 describe-instances) matches the value presented in the console, but the resourcegroupstaggingapi over-reports the number of resources. For ELB listeners/rules, I just did manual inspection in the console.

[RyanFitzSimmonsAK]

Hi @j-land, thank you for your patience. I did some more testing, and I'm still only able to reproduce this issue for some of the resources you listed. For the following resources, they were all absent from both GetResources and their respective describe operations immediately after deletion.

11:51 am - deleted hosted zone 11:51 am - hosted zone absent from ListHostedZones 11:51 am - hosted zone absent from GetResources

12:16 pm - deleted network interface 12:16 pm - network interface absent from DescribeNetworkInterfaces 12:16 pm - network interface absent from GetResources

12:55 pm - deleted elastic load balancer listener 12:55 pm - elastic load balancer listener absent from DescribeListeners 12:55 pm - elastic load balancer listener absent from GetResources

1:02 pm - deleted elastic load balancer listener rules 1:02 pm - elastic load balancer listener rules absent from DescribeListeners 1:02 pm - elastic load balancer listener rules absent from GetResources

The following resources were not immediately deleted from GetResource

12:19 pm - deleted volume 12:19 pm - volume absent from DescribeVolumes 3:01 pm - volume absent from GetResources

11:41 am - deleted ec2 instance 12:51 pm - ec2 instance absent from DescribeInstances 3:40 pm - still visible in GetResources

1:42 pm - deleted EC2 Fleet 3:40 pm - still visible in GetResources and DescribeFleets

I am aware that EC2 handles tagging clean-up in a different way than most services, which makes sense given these observations. I'm going to reach out to the service team about this behavior. In the meantime, as this is a service team issue (not an issue with the CLI), I'm going to move this ticket to our cross-SDK repository for tracking. I'll leave any updates in this issue. Thanks!

Ticket # for internal use : P112989829

[RyanFitzSimmonsAK]

For EC2 resources, tags on deleted resources can take several hours before being properly deleted and removed from GetResources responses. The team is aware of this behavior, and is working on a solution, but I can't promise any specific timeline. In the meantime, if you need stale tags removed at any point, you can reach out, as the service team is able to manually remove stale tags on their end.

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.