Add mapper for AWS Config Rules

[jkufro]

Adds aws_config_mapper. This pulls Ruby AWS SDK data to translate AWS Config Rule results into HDF format json to be viewable in Heimdall. Our intended use case is to incorporate this into an AWS lambda that will pull & translate config compliance and push that up to a Heimdall server via the API.

Example execution (relies on having AWS credentials set up on your local machine): heimdall_tools aws_config_mapper -m my_addl_rule_mappings.csv -o aws_config_results.json

[aaronlippold]

if possible, could we add a function to also add the impact to the control as well? Our impacts are in the range: This will help the overall display and completeness of the data.

0.0 none
0.1 - 0.3 low ( defaults to 0.3 )
0.4 - 0.6 medium ( defaults to 0.5 )
0.7 - 0.8 high ( defaults to 0.7 )
0.9 - 1.0 critical ( defaults to 0.9 )

i would also love to see the lambda work you guys are doing - as we are also working on a lambda task - https://github.com/mitre/serverless-inspec

[jkufro]

if possible, could we add a function to also add the impact to the control as well? Our impacts are in the range: This will help the overall display and completeness of the data.

0.0 none
0.1 - 0.3 low ( defaults to 0.3 )
0.4 - 0.6 medium ( defaults to 0.5 )
0.7 - 0.8 high ( defaults to 0.7 )
0.9 - 1.0 critical ( defaults to 0.9 )

i would also love to see the lambda work you guys are doing - as we are also working on a lambda task - https://github.com/mitre/serverless-inspec

@aaronlippold Are you requesting something different than the function that is defined here and is called here? I have it set up now so that you can tag an AWS Config Rule with Info, Low, Medium, High, or Critical. Should I add the ability to assign a numerical value between 0 and 1 in addition to this?

Also, totally happy to collaborate on lambda work.

[rbclark]

I've already reviewed this, it is just waiting on a final look from @rx294 before being merged.

[rx294]

@jkufro The current design of the aws_config_mapper requires user setup (ie tag placement) on the aws_config rules, which will be a deviation from pattern used in other mappers. Please see below suggestions to align the aws_config_mapper to the general pattern in used in heimdall_tools.

This will help with reusability and consistency of the HDFs generated by aws_config_mapper across multiple users.

1) Control Title mapped title of the AWS_Config rule (as is the case currently) 2) Description: Parsed from the aws_config rule description feild.

3) Check Text to include the config rule arn and parameters used for the rule. Consider formatting the data fields with linefeeds.

4) Set the severity to the "average" of 0.5 (medium)" for all controls.

5) Nist Tag: Create a CSV file mapping config_rule titles to NIST 800-53 mappings as in examples here https://github.com/mitre/heimdall_tools/blob/master/lib/data/owasp-nist-mapping.csv and use it in mapping NIST tags to the controls. This way mapping to NIST controls stays consistant and centralized at heimdall_tools codebase.

6) Remove references to cce_id cis_impact cis_rid fix rationale cweid confidence cis_level at this iteration.

6) Please add sample hdf jsons to this location https://github.com/mitre/heimdall_tools/tree/master/sample_jsons 7) Please add readme updates 8) Please resolve merge conflicts.

[aaronlippold]

if possible, could we add a function to also add the impact to the control as well? Our impacts are in the range: This will help the overall display and completeness of the data.

0.0 none
0.1 - 0.3 low ( defaults to 0.3 )
0.4 - 0.6 medium ( defaults to 0.5 )
0.7 - 0.8 high ( defaults to 0.7 )
0.9 - 1.0 critical ( defaults to 0.9 )

i would also love to see the lambda work you guys are doing - as we are also working on a lambda task - https://github.com/mitre/serverless-inspec

@aaronlippold Are you requesting something different than the function that is defined here and is called here? I have it set up now so that you can tag an AWS Config Rule with Info, Low, Medium, High, or Critical. Should I add the ability to assign a numerical value between 0 and 1 in addition to this?

Also, totally happy to collaborate on lambda work.

@rx294 @rbclark @ejaronne and @aaronlippold meet earlier today and @rx294 will be providing some feedback on our suggested direction for the PR.

Great start so far, but we have a few thoughts on some backend learning we have done but not yet put forward on a docs page about 'how to develop mappers', 'what and how you should transform data', 'do and don't on data transform assumptions', 'what to embed, what to hardcode, and what to reference' etc. all things we should have documented.

The list above I think would be great sections on a 'contributing_a_mapper.md' which may come out of our collaboration on this PR.

Looking forward to it.

[aaronlippold]

@jkufro some additions to the unit and functional tests as well with some representative source and expected converted data would be great as well so we keep the code coverage up to par.

[jkufro]

@jkufro The current design of the aws_config_mapper requires user setup (ie tag placement) on the aws_config rules, which will be a deviation from pattern used in other mappers. Please see below suggestions to align the aws_config_mapper to the general pattern in used in heimdall_tools.

This will help with reusability and consistency of the HDFs generated by aws_config_mapper across multiple users.

1. Control Title mapped title of the AWS_Config rule (as is the case currently)
2. Description: Parsed from the aws_config rule description feild.

1. Check Text to include the config rule arn and parameters used for the rule. Consider formatting the data fields with linefeeds.

1. Set the severity to the "average" of 0.5 (medium)" for all controls.
2. Nist Tag: Create a CSV file mapping config_rule titles to NIST 800-53 mappings as in examples here https://github.com/mitre/heimdall_tools/blob/master/lib/data/owasp-nist-mapping.csv and use it in mapping NIST tags to the controls. This way mapping to NIST controls stays consistant and centralized at heimdall_tools codebase.
3. Remove references to cce_id cis_impact cis_rid fix rationale cweid confidence cis_level at this iteration.
4. Please add sample hdf jsons to this location https://github.com/mitre/heimdall_tools/tree/master/sample_jsons
5. Please add readme updates
6. Please resolve merge conflicts.

@rx294 One major aspect of our use case is continued development/implementation of custom AWS Config rules AWS Docs. My concern with hardcoding Config Rule Name -> 800-53 Control mappings inside the heimdall_tools repository is that as we (or others) develop new rules, we would be required to update the heimdall_tools code to perform proper mappings. Another issue with adding custom rule names to a heimdall_tools CSV mapper is that the source code for each custom rule may not be available open source, so the heimdall_tools CSV for AWS Config could become bloated with rules that only apply to a handful of projects. The original reason I chose to avoid a CSV when creating this mapper was to allow the simplicity of creating a rule with the proper tags to be the only thing necessary to fully configure a rule for mapping & use within heimdall.

EDIT: Here are some examples of Python custom rules made by AWS: https://github.com/awslabs/aws-config-rules/tree/master/python

[jkufro]

@aaronlippold @rbclark @rx294 Please review the latest updates that address the requested changed. Thanks!

[rx294]

LGTM, thanks @jkufro . Please resolve conflicts, that should be the last fix.

[aaronlippold]

Just a reminder, please add an issue to the board to revisit the issue of 'user provided' data - aka overlay data - in our workflow so we can make sure to address the capability we didn't want to try to push in v1.0

[rx294]

Just a reminder, please add an issue to the board to revisit the issue of 'user provided' data - aka overlay data - in our workflow so we can make sure to address the capability we didn't want to try to push in v1.0

https://github.com/mitre/heimdall_tools/issues/73

[aaronlippold]

👍🏻 great job all