[DevOps] Build an inventory of all AWS resources in `vaec-cms`

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:

https://depo-platform-documentation.scrollhelp.site/index.html

283 stars 204 forks source link

[DevOps] Build an inventory of all AWS resources in `vaec-cms` #86930

Closed ndouglas closed 4 months ago

ndouglas commented 4 months ago

Description

We need to dump some report that we can then check to ensure we're doing something with everything.

Tasks

[x] [DevOps] Figure out how to dump this inventory.
[x] [DevOps] Dump this inventory.
[x] [DevOps] Figure out what to do next based on size/complexity/etc. Might require preprocessing, might require secure handling, might require .... ???

Acceptance Criteria

[x] Some inventory of AWS resources has taken some form.
[x] Followup ticket(s) opened as necessary to make that somewhat more useful, deal with it, or something. We don't know what the next step will be next.

olivereri commented 4 months ago

Trying out a few AWS inventory scripts hasn't proven very successful. They either don't work or the data output is difficult to work with. AWS Config in vaec-cms was preconfigured to record resources and are stored in a SQL-like database.

Doing a select * from the web console query tool yielded 2242 resources but more importantly 81 unique resource types. The list of resource types is being reduced to just the resource types that are within project control. Other resources are the sole responsibility of VAEC in the shared administration model.

olivereri commented 4 months ago

I've opened an issue with aws-auto-inventory to address the exception that occurs when writing data. I've patched the script and will create a PR soon. I was able to successfully scan the vaec-cms account and the output is stored in S3:

https://us-gov-west-1.console.amazonaws-us-gov.com/s3/object/aws-auto-inventory?region=us-gov-west-1&prefix=aws-auto-inventory.tar