search

bridgecrewio / checkov

Prevent cloud misconfigurations and find vulnerabilities during build-time in infrastructure as code, container images and open source packages with Checkov by Bridgecrew.
https://www.checkov.io/
Apache License 2.0
7.15k stars 1.12k forks source link

Inconsistent output for identical code scan #4826

Open nicholas-marchini opened 1 year ago

nicholas-marchini commented 1 year ago

Describe the issue I have been running checkov locally (installed on MAC) and in the docker container but getting different results each time I can the exact same code.

The output below is for 4 executions of Checkov on the exact same code but with 3 different results. This run was just using custom checks only. 


┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪  docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --config-file /tf/config/checkov/config.yaml  --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-check CKV2_AWS*,CKV_AWS*
terraform scan results:
Check: CKV3_AWS_IT_DATA_TAGS_test1: "Check that all resources are tagged with the key - slz:test1 and have a valid value"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test2: "Check that all resources are tagged with the key - slz:test2 and have a valid value"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test4: "Check that all resources are tagged with the key - slz:test4 and have a valid value"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250

Passed checks: 4, Failed checks: 4, Skipped checks: 0

┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪  docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --config-file /tf/config/checkov/config.yaml  --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-check CKV2_AWS*,CKV_AWS*
terraform scan results:
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
    FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52

Passed checks: 7, Failed checks: 1, Skipped checks: 0

┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪  docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --config-file /tf/config/checkov/config.yaml  --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-check CKV2_AWS*,CKV_AWS*
terraform scan results:
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
    FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52

Passed checks: 7, Failed checks: 1, Skipped checks: 0

┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪  docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --config-file /tf/config/checkov/config.yaml  --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-check CKV2_AWS*,CKV_AWS*
terraform scan results:
Check: CKV3_AWS_IT_DATA_TAGS_test1: "Check that all resources are tagged with the key - slz:test1 and have a valid value"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test2: "Check that all resources are tagged with the key - slz:test2 and have a valid value"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
    FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV3_AWS_IT_DATA_TAGS_test4: "Check that all resources are tagged with the key - slz:test4 and have a valid value"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250

Passed checks: 11, Failed checks: 5, Skipped checks: 0

If I exclude the custom checks and don't use the config.yaml, so just a normal terraform run with CLI switches then I still get inconsistent results between runs. 

┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪  docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds  --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars  --skip-download --summary-position bottom --download-external-modules true --framework terraform --compact --quiet
terraform scan results:
Check: CKV_AWS_109: "Ensure IAM policies does not allow permissions management / resource exposure without constraints"
    FAILED for resource: aws_iam_policy_document.rds_kms_policy
    File: /data.tf:54-133
Check: CKV_AWS_111: "Ensure IAM policies does not allow write access without constraints"
    FAILED for resource: aws_iam_policy_document.rds_kms_policy
    File: /data.tf:54-133
Check: CKV_AWS_226: "Ensure DB instance gets all minor upgrades automatically"
    FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
    Calling File: /team3.tf:5-84
Check: CKV_AWS_118: "Ensure that enhanced monitoring is enabled for Amazon RDS instances"
    FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
    Calling File: /team3.tf:5-84
Check: CKV_AWS_226: "Ensure DB instance gets all minor upgrades automatically"
    FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
    Calling File: /team6.tf:5-85
Check: CKV_AWS_118: "Ensure that enhanced monitoring is enabled for Amazon RDS instances"
    FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
    Calling File: /team6.tf:5-85
Check: CKV_AWS_226: "Ensure DB instance gets all minor upgrades automatically"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
    Calling File: /team5.tf:5-84
Check: CKV_AWS_118: "Ensure that enhanced monitoring is enabled for Amazon RDS instances"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
    Calling File: /team5.tf:5-84
Check: CKV_AWS_226: "Ensure DB instance gets all minor upgrades automatically"
    FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
    Calling File: /openbanking_pf.tf:5-85
Check: CKV_AWS_118: "Ensure that enhanced monitoring is enabled for Amazon RDS instances"
    FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster_instance.cluster_instance
    File: /../../modules/rds/main.tf:215-250
    Calling File: /openbanking_pf.tf:5-85
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team2_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team1_playground_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team1_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team4_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team2_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team1_playground_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team1_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team4_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52

Passed checks: 41, Failed checks: 26, Skipped checks: 0

┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪  docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds  --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars  --skip-download --summary-position bottom --download-external-modules true --framework terraform --compact --quiet
terraform scan results:
Check: CKV_AWS_109: "Ensure IAM policies does not allow permissions management / resource exposure without constraints"
    FAILED for resource: aws_iam_policy_document.rds_kms_policy
    File: /data.tf:54-133
Check: CKV_AWS_111: "Ensure IAM policies does not allow write access without constraints"
    FAILED for resource: aws_iam_policy_document.rds_kms_policy
    File: /data.tf:54-133
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team1_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team2_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team4_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
    FAILED for resource: module.rds_team1_playground_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team1_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team2_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team4_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
    FAILED for resource: module.rds_team1_playground_team_encrypted.aws_rds_cluster.cluster
    File: /../../modules/rds/main.tf:6-52

Passed checks: 37, Failed checks: 18, Skipped checks: 0

Additional context This happens if I run the commands directly on the MAC and not within the docker container.

nicholas-marchini commented 1 year ago

Updated to latest version 2.3.150 today and it's better, as in most consecutive runs are the same, 1 out of 5 are different.

gruebel commented 1 year ago

hey @nicholas-marchini thanks for reaching out.

It looks like the inconsistency comes from the massive usage of the same module rds not so surprising for me. Additionally using multiple tfvars files doesn't make it easier.

nicholas-marchini commented 1 year ago

@gruebel Thanks for the reply. We operate a multi-tenent AWS account and do have the need to use multiple tfvars files right now.

stale[bot] commented 1 year ago

Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at https://slack.bridgecrew.io Thanks!

nmarchini commented 1 year ago

This is still an issue. What can be done to resolve it?

gruebel commented 1 year ago

hey @nmarchini

Do you have the same setup, multiple times using the same module? If yes, how many times is it?

nmarchini commented 1 year ago

@gruebel We call the module 9 times. I have used the latest docker image and am still getting inconsistent results

nmarchini commented 1 year ago

Any way we can progress this please?

novekm commented 10 months ago

Same issue here - different tests are running on my local machine (Mac) than in a container AWS CodeBuild is managing (more tests are checked in the container). However in my case the issue is happening even within just a single module (not referencing any other modules).

Is there way to define/enforce a list of tests that you want checkov to run? Something like checkov -d . --run-tests: 'CVE_AWS'? How does checkov determine what tests to run when presented a file or directory?

nmarchini commented 9 months ago

I've given up on checkov as having this issue open since Apr 2, 2023 is poor, since Checkov got bought the level of response and interaction to issues posted here has dropped drastically. We moved to TFsec now and are very happy with it.

stale[bot] commented 3 months ago

Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at codifiedsecurity.slack.com Thanks!