Store and return last failure from validate vs exit on first failure

[davidalger]

This PR accomplishes two things:

• Cleans up output on terraform-validate adding informative marker indicating directory in which a failure occurred, which is crucial when running this in CI against a repository with a dozen or more modules.
• Updates terraform-validate to correctly check each argument passed in by pre-commit rather than exiting on the first failure encountered.

[davidalger]

Here's what I did to test this:

a) Added the following to .pre-commit-config.yaml and then run pre-commit install in a local checkout of a terraform module:

repos:
  - repo: https://github.com/davidalger/pre-commit
    rev: "a787da3a8"
    hooks:
      - id: terraform-fmt
      - id: terraform-validate

Can also be done via HEAD referencing local checkout:

repos:
  - repo: ~/Work/pre-commit
    rev: "HEAD"
    hooks:
      - id: terraform-fmt
      - id: terraform-validate

b) Run pre-commit on all files with intentionally bad code:

$ pre-commit run --all-files
Terraform fmt............................................................Passed
Terraform validate.......................................................Failed
- hook id: terraform-validate
- exit code: 1

--> Running 'terraform validate' in directory '.'

Initializing provider plugins...
- Using previously-installed hashicorp/aws v3.20.0
- Using previously-installed hashicorp/template v2.2.0

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/aws: version = "~> 3.20.0"
* hashicorp/template: version = "~> 2.2.0"

Terraform has been successfully initialized!
Success! The configuration is valid.

--> Running 'terraform validate' in directory 'examples/cloud-init-users'
Initializing modules...
Downloading terraform-aws-modules/vpc/aws 2.33.0 for vpc...
- vpc in .terraform/modules/vpc

Error: Unsupported Terraform Core version

  on .terraform/modules/vpc/versions.tf line 2, in terraform:
   2:   required_version = "~> 0.12.6"

Module module.vpc (from terraform-aws-modules/vpc/aws) does not support
Terraform version 0.13.5. To proceed, either choose another supported
Terraform version or update this version constraint. Version constraints are
normally set for good reason, so updating the constraint may lead to other
errors or unexpected behavior.

Error: Unsupported Terraform Core version

  on .terraform/modules/vpc/versions.tf line 2, in terraform:
   2:   required_version = "~> 0.12.6"

Module module.vpc (from terraform-aws-modules/vpc/aws) does not support
Terraform version 0.13.5. To proceed, either choose another supported
Terraform version or update this version constraint. Version constraints are
normally set for good reason, so updating the constraint may lead to other
errors or unexpected behavior.

--> Running 'terraform validate' in directory 'examples/simple-app-server'
Initializing modules...

Initializing provider plugins...
- Using previously-installed hashicorp/random v3.0.0
- Using previously-installed hashicorp/aws v3.20.0
- Using previously-installed hashicorp/template v2.2.0

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/random: version = "~> 3.0.0"
* hashicorp/template: version = "~> 2.2.0"

Terraform has been successfully initialized!
Success! The configuration is valid.

Failed as expected with errors found in one of the validations.

c) Attempt to commit the bad code:

$ git commit -am "Test bad commit"
Terraform fmt............................................................Passed
Terraform validate.......................................................Failed
- hook id: terraform-validate
- exit code: 1

--> Running 'terraform validate' in directory 'examples/cloud-init-users'
Initializing modules...

Error: Unsupported Terraform Core version

  on .terraform/modules/vpc/versions.tf line 2, in terraform:
   2:   required_version = "~> 0.12.6"

Module module.vpc (from terraform-aws-modules/vpc/aws) does not support
Terraform version 0.13.5. To proceed, either choose another supported
Terraform version or update this version constraint. Version constraints are
normally set for good reason, so updating the constraint may lead to other
errors or unexpected behavior.

Error: Unsupported Terraform Core version

  on .terraform/modules/vpc/versions.tf line 2, in terraform:
   2:   required_version = "~> 0.12.6"

Module module.vpc (from terraform-aws-modules/vpc/aws) does not support
Terraform version 0.13.5. To proceed, either choose another supported
Terraform version or update this version constraint. Version constraints are
normally set for good reason, so updating the constraint may lead to other
errors or unexpected behavior.

Failed as expected.

d) Commit some passing changes to verify it passes:

$ git commit -am "Test good commit"
Terraform fmt............................................................Passed
Terraform validate.......................................................Passed
[master 56b151f] Test good commit
 1 file changed, 1 insertion(+)

e) Run pre-commit on all files again expecting no failures:

$ pre-commit run --all-files
Terraform fmt............................................................Passed
Terraform validate.......................................................Passed

[brikis98]

Tests passed! Merging now.

[brikis98]

Ah, I just realized we should apply similar changes to the terraform-fmt hook too: https://github.com/gruntwork-io/pre-commit/pull/46

[brikis98]

https://github.com/gruntwork-io/pre-commit/releases/tag/v0.1.12