search

iacsecurity / tool-compare

MIT License
276 stars 59 forks source link

Add dynamic / live test cases #21

Open yi2020 opened 3 years ago

yi2020 commented 3 years ago

Over the past year, infrastructure-as-code security began to evolve to look beyond just the code itself. We've seen this with Accurics, Bridgecrew, Fugue and Indeni Cloudrail's offerings. We also saw this recently with driftctl's launch.

An IaC security tool comparison needs to take this into account. This means we need to create a staging AWS environment that can be used in conjunction with the IaC security scans to show the capabilities beyond just static analysis.

We should figure out a way to pull this information into this tool-compare repository in a manner that's reproducible to anyone who seeks to do so.

yi2020 commented 3 years ago

@owenrumney @ismailyenigul @kaplanlior @JamesWoolfenden - as you all have engaged with this repo, would love to hear your thoughts. Anyone else is also welcome to jump in on this conversation.

yi2020 commented 3 years ago

Update to this issue: The best idea I have for this is to create an AWS account (and later azure, etc.) and then:

  1. Create an IAM user with ReadOnlyAccess, and post the accesskey and secret in this repo (so there's no need for people to supply their own in run_all_tools).
  2. In that account, simulate various issues that don't run a bill, using Terraform code.
  3. Add the account to all tools that support it (I know of Accurics, Bridgecrew, Fugue and Indeni Cloudrail). Focus is only on tools that do IaC security and show the results in relation to IaC (so no generic CSPM tools). Hopefully each tool can have a read-only user generated so we can post the credentials here without issue.
  4. Track which tools find the issues simulated in #2 above.

NOTE: Write credentials to the AWS account and the various tools will be saved in the secrets of this repo, accessible only to those individuals who are designated (ideally, one representative from each tool).

If anyone has feedback on this approach, would love to hear it :)

yi2020 commented 3 years ago

This also makes me wonder - some of the open source tools don't yet look at live accounts, so need to think of how to separate the measurement between those that do and those that don't.