Ingest AWS tags using the resourcegroupstaggingapi

[achantavy]

Feature request template

Title: Ingest AWS tags using the resourcegroupstaggingapi

Description:

Describe your idea. Please be detailed. If a feature request, please describe the desired behavior, what scenario it enables, and how it would be used.

We would like to add AWS tags to the graph. #174 is the first attempt to do this, but it requires each asset sync to handle tags themselves.

It looks like we might be able to ingest all AWS resource tags with just the resourcegroupstaggingapi.

I started on a prototype:

import logging
from string import Template
from cartography.util import run_cleanup_job
logger = logging.getLogger(__name__)

def get_tag_data(boto3_session, region):
    """
    Create boto3 client and retrieve tag data.
    """
    client = boto3_session.client('resourcegroupstaggingapi', region=region)
    paginator = client.get_paginator('get_resources')
    resources = []
    for page in paginator.paginate(
        # Only ingest tags for resources that Cartography supports.
        # This is just a starting list; there may be others supported by this API.
        ResourceTypeFilters = [
            # 'acm',
            # 'backup',
            # 'cloudfront',
            # 'cloudwatch',
            # 'datapipeline',
            'ec2',
            # 'elasticache',
            # 'elasticfilesystem',
            # 'elasticloadbalancing',
            # 'elasticmapreduce',
            'es',
            # 'events',
            # 'firehose',
            # 'kinesis',
            # 'lambda',
            'rds',
            # 'redshif$',
            's3',
            # 'sagemaker',
            # 'sqs',
            # 'transfer'
        ]
    )        resources.extend(page['ResourceTagMappingList'])
    return {'ResourceTagMappingList': resources}

def load_tag_data(neo4j_session, data, region, current_aws_account_id, aws_update_tag):
    INGEST_TAG_TEMPLATE = Template("""
    MATCH (resource:$resource_label{id:{ResourceId}})
    MERGE(aws_tag:AWSTag:Tag{id:{}})
    MERGE (resource)-[r:TAGGED]->(aws_tag)
    SET r.lastupdated = {UpdateTag}, r.first
    """)

[optional Relevant Links:]

Any extra documentation required to understand the issue.

• https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/Welcome.html
• https://github.com/lyft/cartography/pull/174#issuecomment-573903312
• https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/resourcegroupstaggingapi.html

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale[bot]]

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new change or reopen this one.