search

iacsecurity / tool-compare

MIT License
276 stars 59 forks source link

Add context-aware tests #37

Closed charlesk92 closed 3 years ago

charlesk92 commented 3 years ago

Most of these checks are simpler. Bridgecrew and Indeni announced support for context-aware rules. Would be good to have more test cases that support context-type issues.

FYI @christophetd

yi2020 commented 3 years ago

@charlesk92 I've been thinking about this and am wondering if such test cases should be separate from the main set of test cases, or included in the main set. For example, there's terraform/aws/iam/iam-entities/human_users_defined. Some of the tools use context-awareness to determine the issue (checkov and cloudrail), while the others don't.

I don't think it matters (to a user) if context-awareness is used to determine an issue, as long as it's found. Therefore, my logic leads me to simply be adding test cases that are relevant for various needs (CIS, PCI, or otherwise) within the main set of test-cases and check which tools catch them, however they do it.

There's a flip side to this - adding cases where the tool should NOT alert. I think that's where context-awareness may come more into the picture. Context awareness can be used to reduce noise. For example, there's terraform/aws/iam/resource-policies/s3_bucket_acl_public_all_users_canned_with_overriding_access_block, where you could say the S3 bucket is not at issue because the public access block overrides it.

But then, there's people who think that not alerting about an S3 bucket being public simply because there's an access block is not the right way to go. They still want to know that the bucket has a public ACL even though it's not in effect.

It's somewhat of an opinionated discussion. Would appreciate the thoughts of others here - @kaplanlior @owenrumney @christophetd and anyone else who wants to add.

charlesk92 commented 3 years ago

hm good point. To me, it was important the rules can factor multiple resources into the business logic. This was an evaluation parameter under the ability to write custom checks: https://blog.christophetd.fr/shifting-cloud-security-left-scanning-infrastructure-as-code-for-security-issues/ To me, this is what matters the most for "context-awareness".

yi2020 commented 3 years ago

I'll close this issue and we'll simply add test cases where relevant based on regulations etc.