search

lyft / cartography

Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.
https://lyft.github.io/cartography/
Apache License 2.0
2.92k stars 325 forks source link

Fetch AWS EC2 AMIs based on what's in the graph #652

Open ryan-lane opened 3 years ago

ryan-lane commented 3 years ago

Description:

Fetching AMIs is difficult, because the default filter options for the API are roughly:

Fetching all images takes nearly a minute per region, per account, and loads tens of thousands of images, which aren't being actively used.

It would be ideal to fetch the AMIs in a multi-step manner, which introspects the graph to determine which AMIs should be fetched:

  1. Load EC2 instances, launch configurations, instance templates, etc.
  2. Query the graph to find AMIs in use
  3. Fetch/load AMIs based on what's in use.

I'm not sure at this point if cartography is doing graph introspection to determine what to fetch at this point, so I wanted to float the idea first.

achantavy commented 3 years ago

I'm not sure at this point if cartography is doing graph introspection to determine what to fetch at this point, so I wanted to float the idea first.

I believe we do this with IAM policies: there are lots of policies that AWS APIs return to us that don't apply to our environment, so we only sync policies that are attached to a principal: https://github.com/lyft/cartography/blob/41425d1727f10af497472346e94d98d0e1ed66e2/cartography/intel/aws/iam.py#L717-L718

So I agree with your proposal.

ryan-lane commented 3 years ago

Awesome. I may take a stab at this soon, then.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

stale[bot] commented 2 years ago

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new change or reopen this one.

ryan-lane commented 2 years ago

Not stale. We're working on this.

chandanchowdhury commented 3 weeks ago

Is this good to close?