search

JohannesEbke / aws_list_all

List all your AWS resources, all regions, all services.
MIT License
414 stars 97 forks source link

initial commit for adding AWS profile support #28

Closed alexjmoore closed 4 years ago

alexjmoore commented 5 years ago

Hi,

Great little tool. I did some work to add support for optionally specifying a particular credential profile with a --profile flag (used -c as shorthand due to clash with existing --parallel option). It also attaches the profile name to the output filename to easily determine which profile was used.

Feel free to adjust/refactor as needed, I did some basic testing and I don't think I've broken anything, but you never know!

Any questions let me know, Alex

alexjmoore commented 5 years ago

I believe the test failures relate to changes in the endpoint_hosts and service_regions cache, I did try to create them but the 'recreate-caches' option doesn't appear to update the json for some reason - I might be barking up the wrong tree there. Also appears comprehend medical now has some listing ops: https://docs.aws.amazon.com/comprehend/latest/dg/API_Operations_AWS_Comprehend_Medical.html

I'm also unclear about listing.py line 56 where boto3.Session is being used directly - wasn't sure enough about what you're doing there, but figure that might warrant a little refactor to use get_client too.

JohannesEbke commented 4 years ago

Hi @alexjmoore , I've first taken up the issue with the endpoint_hosts and service_regions cache, and hopefully clarified their use - they were first meant to be just local caches for each user. I've now added a flag --update-packaged-values that actually overwrites the resource files where they are, so that it's easier to get out of this situation: https://github.com/JohannesEbke/aws_list_all/compare/505efa3..1600bbe0e3206f8758cb7025c80b95e0183bdab2

alexjmoore commented 4 years ago

I see it is still failing tests due to the comprehendmedical endpoint, thoughts?

gloria-willadsen-brandorr commented 4 years ago

Please merge this into master. It is incredibly useful.

sergray commented 4 years ago

It is possible to specify the profile for AWS CLI with AWS_PROFILE environment variable.

Am I missing the use-case for specifying the profile as CLI argument?

alexjmoore commented 4 years ago

AWS_PROFILE specifies the default profile to use, of course you could wrap the aws_list_all with something that sets the variable to a specific profile before calling, but in my personal opinion it is better to explicitly select the profile in the tool itself so you have both options, this is also consistent with how the AWS CLI works:

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

Ultimately if you are dealing with many different accounts with unique credentials it is more convenient to be able to specify with a command line option rather than changing the env each time.

JohannesEbke commented 4 years ago

Thanks @alexjmoore for providing this PR! I've merged it with one alteration: The default profile is set to None; as this frees up boto3 to pick credentials which are not managed via profile (otherwise using tools that provide credentials as environment variables like https://github.com/99designs/aws-vault will break).

I am a bit unhappy about all files now ending with "_None.json", and I played around with e.g. putting all profile files into one directory named as the profile (similar to -d) but ended up not doing it, since then it would be dangerous to e.g. move all files into one directory without clobbering some of them. And just dropping the profile from the filename would be inconsistent...