[Bug] unable to unmarshal conftest output or Anyone able to approve Atlantis policy failures

kumaresh0 commented 7 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Anyone ( non policy owners ) able to approve Atlantis policy failures ( seems major bug ) I am using below server config with custom_policy_check: true & policy_check: true
if I use custom_policy_check: flase & policy_check: true i get the unable to unmarshal conftest output error

But based on this documentation https://www.runatlantis.io/docs/policy-checking.html#step-2-define-the-policy-configuration we tried the config as i mentioned below

Reproduction Steps

Nothing special just used Alpine image and added the below server-side workflow and repo side workflow then triggered policy failure, if anyone comments the atlantis approve_policies policy failures are fixed with approval

Logs

Issue screen shot

Environment details

If not already included, please provide the following:

Atlantis version: v0.27.1
Deployment method: ecs
If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: Yes ( it' already latest release )

Atlantis flags: No custom flags ( using default flags with repo details )
ECS environment variables

container_definition_environment = [
{
  name  = "ATLANTIS_CHECKOUT_STRATEGY"
  value = var.checkout_strategy
},
{
  name  = "ATLANTIS_ALLOW_COMMANDS"
  value = var.atlantis_allow_commands
},
{
  name  = "ATLANTIS_ALLOW_REPO_CONFIG"
  value = var.allow_repo_config
},
{
  name  = "ATLANTIS_RESOURCE_BUCKET_NAME"
  value = data.aws_s3_bucket.atlantis_bucket.id

},
{
  name  = "ATLANTIS_LOG_LEVEL"
  value = var.atlantis_log_level
},
{
  name  = "ATLANTIS_ASSUME_ROLE_NAME"
  value = var.atlantis_assume_role_name
},
{
  name  = "ATLANTIS_PORT"
  value = var.atlantis_port
},
{
  name  = "ATLANTIS_CONFIG"
  value = var.atlantis_config
},
{
  name  = "ATLANTIS_ATLANTIS_URL"
  value = local.atlantis_url
},
{
  name  = "ATLANTIS_WRITE_GIT_CREDS"
  value = true
},
{
  name  = "ATLANTIS_GH_USER"
  value = var.atlantis_github_user
},
{
  name  = "ATLANTIS_REPO_ALLOWLIST"
  value = join(",", var.atlantis_repo_allowlist)
},
{
  name  = "ATLANTIS_REPO_CONFIG"
  value = var.atlantis_repo_config
},
{
  name  = "ATLANTIS_ENABLE_POLICY_CHECKS"
  value = var.atlantis_enable_policy_checks
},
{
  name  = "ATLANTIS_HIDE_PREV_PLAN_COMMENTS"
  value = var.atlantis_hide_prev_plan_comments
},
{
  name  = "ATLANTIS_DISABLE_APPLY"
  value = var.atlantis_disable_apply
},
{
  name  = "ATLANTIS_DEFAULT_TF_VERSION"
  value = var.atlantis_default_terraform_version
},
{
  name  = "ATLANTIS_POLICY_DIRECTORY"
  value = var.atlantis_policy_directory
},
{
  name  = "ATLANTIS_POLICY_REPO_NAME"
  value = var.atlantis_policy_repo_name
},
{
  name  = "ATLANTIS_POLICY_REPO_REF"
  value = var.atlantis_policy_repo_ref
}
]

Atlantis server-side config file:

repos:
- id: "/.*/"
  branch: "/master/"
  pre_workflow_hooks:
    - description: Pull the Atlantis policies.
  post_workflow_hooks:
    - run: python3 /home/atlantis/review.py 
  allow_custom_workflows: false
  custom_policy_check: true
  policy_check: true
  allowed_overrides: [workflow, apply_requirements, delete_source_branch_on_merge]
  apply_requirements: [approved, mergeable]
  import_requirements: [approved, mergeable]

metrics:
  prometheus:
    endpoint: /metrics

policies:
    conftest_version: 0.45.0
    owners:
      users:
        - user1
        - user2
        - user3
    policy_sets:
        - name: audit_policy
          path: /home/atlantis/policy/security
          source: local
          approve_count: 2
          owners:
            users:
              - securityuser1
        - name: comman_policy
          path: /home/atlantis/policy/regula
          source: local

workflows:
  default:
    plan:
      steps:
      - init:
          extra_args: [ "-reconfigure"]
      - run: TF_WORKSPACE=$WORKSPACE tflint --config /home/atlantis/.tflint.hcl
      - plan
      - run: terraform$ATLANTIS_TERRAFORM_VERSION show -no-color -json $PLANFILE > tfplan.json
    apply:
      steps:
      - apply
    policy_check: &policy_check
      steps:
      - run: |
          aws sts get-caller-identity --output json | jq '{"aws": .}' | jq '{"external": .}' > external-data.json
      - run: conftest test tfplan.json --namespace main --namespace security -o table --policy /home/atlantis/policy -d external-data.json
    import:
      steps:
      - import

Repo atlantis.yaml file:

version: 3
automerge: true
delete_source_branch_on_merge: true

projects:
- name: test-case/default-workflow
  dir: test-case/default-workflow
  workspace: default
  autoplan:
    when_modified: ["*.tf", "../modules/**.tf", "Terrafile"]
    enabled: true
  workflow: default

Our Atlantis is deployed in ECS fargate with ghcr.io/runatlantis/atlantis:v0.27-alpine

Additional Context

kumaresh0 commented 6 months ago

similar issuer reported here : https://github.com/runatlantis/atlantis/issues/4308

kumaresh0 commented 6 months ago

Adding some update on this:

From console, i can see policy test results

example

Resources: {"aws_security_group.inline_invalid_security_group"}

4 tests, 2 passed, 0 warnings, 2 failures, 0 exceptions

But in PR it show unable to unmarshal conftest output and anyone able to approve the policy failures

runatlantis / atlantis