policy_check: false not working with v0.28.5

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

We've found that there as been added a new feature to atlantis related to contest policies that allows to enable policies only in some repositories or specific directories (by disabling those directories that we don't want to run policies on).

See https://github.com/runatlantis/atlantis/pull/3503

we've updated to version v0.28.5 and tried this feature to disable policy checks in all projects (directories) except one (for testing and debugging) but it seems like Atlantis is running policy checks in all directories/projects anyway.

Not sure how to debug and fix it.

Reproduction Steps

We ar running v0.28.5 We have something like this in our atlantis.yaml config in certain repository:

projects:
- name: sandbox
  dir: terraform/sandbox
  workflow: sandbox
  apply_requirements: []
  policy_check: true

- name: staging
  dir: terraform/staging
  workflow: staging
  policy_check: false

The policy checks are being executed in sandbox but also in staging and other projects with policy_check set to false :(

Logs

Nothing interesting in logs

{"level":"info","ts":"2024-07-19T09:30:25.830Z","caller":"server/server.go:184","msg":"Policy Checks are enabled","json":{}}
{"level":"info","ts":"2024-07-19T09:30:26.428Z","caller":"policy/conftest_client.go:153","msg":"failed to get default conftest version. Will attempt request scoped lazy loads DEFAULT_CONFTEST_VERSION not set","json":{}}
...
{"level":"info","ts":"2024-07-19T09:30:26.428Z","caller":"server/server.go:1017","msg":"Atlantis started - listening on port 4141","json":{}}
....

Environment details

Ideas on how to debug or where the failure could be?

Thanks in advance!

runatlantis / atlantis