Terraform unit testing framework

[alexharv074]

This may be a duplicate of https://github.com/hashicorp/terraform/issues/5059 but the release of Terraform 0.12 with iteration features has made the need for a real Terraform unit testing framework more urgent. In other words, there should be a Terraform equivalent of Puppet's rspec-puppet.

[alexharv074]

@apparentlymart , I have been reviewing the comments above carefully to understand why we ended up introducing the limitation that testing occurs at the level of the resource rather than module - a key departure from how Rspec-puppet works. At the moment, this isn't necessarily as big a deal as it might be, simply because, unlike Puppet, Terraform has no control flow (if statements, each loops etc) that can operate at this level. (Is there any chance that this could ever change?)

Even so, I am still not convinced about the reasoning given above:

I previously was thinking about doing this at the whole-module level, but I think in practice that would lead us back to my more recent idea of writing test doubles for all of the providers, because I think fake static data would not be sufficient in most real-world cases.

I am a little confused by this because nearly all unit testing involves passing fake static data into the code under test. Occasionally you want real data and/or dynamic data but those would be the edge cases.

This leaves me wondering if the wrong choice was made above, and if testing at the level of the module is in fact the right approach? It certainly seems more intuitive to me.

So I wonder: how hard would it be to adjust "testing eval" to allow the whole module to be tested?

[OJFord]

Good discussion here! I've also wanted this for modules in the past; just to offer a defence for HCL, I imagined it being something like:

# mymodule/usage-1.tftest

module "mymodule" {
  usage = 1
  params = "that are under test"
  # ...
}

resource "foo" "bar" {
  # the expected declaration given inputs above
}

# ...

i.e. using existing syntax but in a tftest file, the module would be the inputs under test, and the resources would be the expected outputs; resource attributes not specified would be 'don't care' values not checked.

[alexharv074]

For the record I have written a blog post about the work I did here in the hope of inspiring others to vote for this feature https://alexharv074.github.io/2019/06/15/adventures-in-the-terraform-dsl-part-vi-towards-a-unit-testing-framework.html

[rismoney]

Found this (supports HCL)

https://github.com/instrumenta/conftest/blob/master/README.md

[OJFord]

Thank you @rismoney that looks fantastic! And more widely useful than just HCL. Copying a bit from the readme for others who, like I did until now, glossed over your link:

conftest is a utility to help you write tests against structured configuration data. For instance you could write tests for your Kubernetes configurations, or Tekton pipeline definitions, Terraform code, Serverless configs or any other structured data. [...] As of today conftest supports:

YAML JSON INI TOML HCL CUE Dockerfile HCL2 (Experimental)

[...] For instance, save the following as policy/deployment.rego:

package main

deny[msg] {
  input.kind = "Deployment"
  not input.spec.template.spec.securityContext.runAsNonRoot = true
  msg = "Containers must not run as root"
}

deny[msg] {
  input.kind = "Deployment"
  not input.spec.selector.matchLabels.app
  msg = "Containers must provide app label for pod selectors"
}

[alexharv074]

@OJFord , I'm sure this is a great tool for automated testing but this thread and issue is about a unit test framework for testing Terraform's evaluation logic.

[OJFord]

I'm not sure in what sense the above example is not a unit test. It doesn't cover every sort of test we'd like for terraform, but it certainly covers some ground.

It's also interesting I think because one way of working a more general solution could be to feed terraform plan output (not possible today, would need a non-binary -out, something new, or to parse the human-readable output) to conftest in order to be able to test on defaults, diffs, implicit dependencies, etc.

[alexharv074]

@OJFord , "Containers must not run as root" unless I'm misreading (I'm not a K8S user) looks like a test on the built system i.e. a system test?

[OJFord]

No, that example is something you'd run over all your k8s *.yaml files (config for applying to the cluster, not the cluster itself), and in each one that has:

kind: Deployment

check that it also has

spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true

[alexharv074]

@OJFord but it's still not a unit test. It's a data test then.

[apparentlymart]

After peeping into the HCL2 implementation, it looks like conftest is evaluating the test cases against a data structure representing the static config, without evaluating any expressions.

I can see that being useful for describing policies for what is allowed or not, but unfortunately I think it's not sufficient for unit testing because it's not aware of HCL's expression language, built in functions, references to other objects, etc.

In principle it could assert against something like the output of the prototype we've been discussing here, but I'm not sure writing the tests in Rego is any better than writing them in e.g. Python or Ruby. Were we to go for a DSL for writing tests, I expect we'd go for something based in the Terraform language itself, since if it's going to be something else it's better for it to be a language folks are likely to already know. 🤔

[eerkunt]

Don't want to create any flame discussion about What the testing should be for terraform ?, but ;

I have been maintaining terraform-compliance | website for a while. It reads terraform plan output and allows you to drill down the resource structure you have in HCL/plan output. Of course, it doesn't have flexibilities of a programming language where you can switch to beast mode quite easily.

Main focus - as it has in its name, is compliance, thus the audience is not only developers but security people who lack of development skills also.

We have been using it in several different enterprises in a CI/CD pipeline where all changes are tested against a remote git repository that has the compliance tests. Working good so far.

[zensolution]

I also worked kind of testing framework for terraform. The most thing I missed is that I am not able to mockup data source, which prevent me from testing modules using data source.

[nhurel]

On my side I've started this project : https://github.com/nhurel/terraspec Still lot of work to do but it should work with any provider. You can go run ../../main.go from the example/aws directory to see what it looks like. Feedback welcomed.

[nathanielks]

That looks promising, thanks @nhurel !

[zensolution]

@nhurel Your repo looks quite good. But unfortunately, I am building stuff in java and need to figure out how to incorporate your work there.

[zioalex]

Hi All! Is it possible to get some more attention from Hashicorp. This to me it's a key feature to enable fast deployment cycle and the real Devops way.

[tomelliot16]

Personally I tried every suggestion in the comments above. I even followed using sentinel to make plan validation seeing that's what hashicorp seems to suggest in their documentation if you are an enterprise customer (Its a trap). After spending good number of hours, I really think terraspec is the solution and it doesn't seem like hashicorp prioritizing a competitor. Therefore I have started working out the features left with @nhurel.

[eugenetaranov]

I initially attempted writing own python tool with help of pytest that would parse HCL and make assertions about resource properties. But got blocked by expressions (terraform functions, ternary operators), they need to be evaluated where possible .. I'll appreciate any hints on this. Meanwhile giving https://github.com/nhurel/terraspec a try.

[omarismail]

Hey y'all!

Omar (PM of Terraform) here 👋 . The team has been looking into this and we have developed early prototypes for new testing capabilities in terraform. If you'd be interested in having a look, please email me at oismail@hashicorp.com and we can set up some time to run through the prototype and discuss the main ideas.

[omarismail]

Hey y'all,

With the new Terraform Test framework coming out in the official 1.6 release right around the corner, I am excited to say that we are already working on follow up improvements from early feedback: Mocking.

We are looking to enable Provider/Module Mocking natively with the Terraform Test framework.

I’d love to share some early prototypes and workflows with you all to get early feedback. If you are interested, please email me at oismail@hashicorp.com and we can schedule a time! Much appreciated!

[apparentlymart]

Terraform v1.7 introduced various features for mocking and overriding provider-based features.

This issue had a pretty broad scope, so it's tough to say whether those new features represent this feature request being "done". I think the best approach then is to close this issue to represent that it's broadly solved, but I invite anyone who was participating here to share specific feedback they have about the details in separate issues, so that we can discuss them in more detail in their own space.

Thanks!

[alexharv074]

@apparentlymart it definitely looks done to me! On behalf of Terraform Testers everywhere, thank you!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.