Enforcing policy checking

[dupuy26]

For Atlantis servers where conftest policy checking is enabled, projects using a terraform_version earlier than 0.12.0 are effectively ignored since there is no way to generate the JSON state file that conftest requires.

There are two possible approaches to eliminating this loophole and enforcing policy checks for all projects. One would be a server configuration setting (flag and environment variable) ‑‑enforce-policy-checks/ATLANTIS_ENFORCE_POLICY_CHECKS that would enable policy checks (just like the existing setting) but change the policy check behavior to fail (rather than succeed) if the terraform version is not 0.12 or later.

Another approach would be a server configuration setting --minimum-tf-version that would be more general, disabling the use (even if explicitly configured) of any Terraform version less than the configured minimum. Setting this value to 0.12 or later would have the same effect w.r.t. policy checking, but this feature could also be used to support an organization's policy for Terraform version support (as well as potential future feature requirements).

Before looking at possible implementations for either of these approaches, I would be very interested in hearing from others about their preference for one or the other.

[nishkrishnan]

I like the second approach (--minimum-tf-version) I think it's reasonable to assume that an organization would not want to support all terraform versions.

[dupuy26]

In the end, we needed to implement a "pre-plan-conftest" script and use it in the plan stage (after terraform init and before terraform plan) to prevent a malicious (internal) PR from using a data "external" block to run arbitrary programs in the Atlantis container.

Once we had that script invoked in every workflow, it was trivial for the script to also check that ATLANTIS_TERRAFORM_VERSION is an acceptable version (in our case 0.12 <= version < 1.2 since a newer version of Terraform might introduce new features that could be exploited).