Feature request : Optional Argument for custom services and custom regions scope for collect command

[rams3sh]

I have been faced with writing custom scripts for collection of various resources across services in AWS such as finding route53 domains and zones, or security groups across ec2 services.

Cloudmapper already has collect command which is used to collect resources. Having customised collect would really help people who want only a specific collection to be done rather than collection of all resources.

The expected command would look something of this kind :-

python3 cloudmapper.py collect --regions us-east-1 us-west1 --services iam route53 ec2 --account my_aws_account

This request also stems from one of the asks from the Cloud Security Channel where the person wanted route53 zones and recordsets to be dumped.

This would be helpful for people who may want to use cloudmapper for it's collection capability. This may impact the existing audit modules which is dependant on the collections , but some collection metadata json can be created, reading which the modules can come to know about the coverage of collection based on which it can give appropriate warning or error to the user mentioning required collection data does not exist for audit to proceed or something of that nature.

[0xdabbad00]

The regions option does exist currently: https://github.com/duo-labs/cloudmapper/blob/5b733b4ccacbb328a2a99de91ab0939888764d76/commands/collect.py#L576

The option to collect only data from specific services would be new. My initial thought was "But what if someone only wants specific data within a service?" but I don't think we need to get that fine-grained, and also that would likely break things, because if someone thought "I only want S3 bucket policies", but you need to also list the buckets, so a service level filter would be best.

As with any customization like this, the other features of CloudMapper will not work as expected, for example, if someone only collects the route53 data and IAM, and then tries to see what EC2s are public, they won't have data for that and the command won't understand that that data was not collected. I think this is. acceptable though.

This should be fairly easy to implement. You would need to add another parameter around here: https://github.com/duo-labs/cloudmapper/blob/5b733b4ccacbb328a2a99de91ab0939888764d76/commands/collect.py#L576

Then just before this line, would check if runner["Service"] is in your allow list, similar to what was done for the region check above that. https://github.com/duo-labs/cloudmapper/blob/5b733b4ccacbb328a2a99de91ab0939888764d76/commands/collect.py#L331

[rams3sh]

@0xdabbad00 Also , would be nice to have a list of supported services being shown for someone to decide what could be the possible values for --services option.

This may not be possible probably with regions since regions are dynamically queried and corresponding data collected, however services are defined as per the collect_commands.yaml. https://github.com/duo-labs/cloudmapper/blob/main/collect_commands.yaml

Adding this here since it is a related feature.

[rams3sh]

You may also have to handle the priority between regions and services.

Global Services such as iam should not honour the regions parameter , whereas regions parameter should only apply to regional services.

Like the following command, the regions parameter has no significance , since iam being a region agnostic service.

python3 cloudmapper.py collect --regions us-east-1 us-west1 --services iam --account my_aws_account

I dont know if you have handled the logic already with existing --regions , but putting it here anyway so that this case is in radar when --services comes into the picture.

[0xdabbad00]

CloudMapper handles the global services by cheating a bit by always ensuring it includes us-east-1 or whatever the default region is for that partition. So for example, if you specified only us-west-2, then the collect would still collect from us-east-1 and us-west-2. https://github.com/duo-labs/cloudmapper/blob/5b733b4ccacbb328a2a99de91ab0939888764d76/commands/collect.py#L223