Validate doesn't work for projects with nested sub-directories

[jamiezieziula]

What problem are you facing?

When running pre-commit run --all-files or when its run automatically on a commit, i get the following result, even when some terraform code is note valid:

Terraform fmt............................................................Passed
Terraform validate.......................................................Passed
Terraform docs...........................................................Passed

My terraform code is nested in various folders throughout my repo, but in this specific case i have the following folder structure:

.
├── LICENSE
├── README.md
├── aws
│   ├── README.md
│   ├── aws-services
│   │   ├── delete-default-vpcs
│   │   │   ├── README.md
│   │   │   ├── data.tf
│   │   │   ├── iam.tf
│   │   │   ├── main.py
│   │   │   ├── main.tf
│   │   │   ├── providers.tf
│   │   │   ├── variables.tf
│   │   │   └── versions.tf
│   │   ├── network
│   │   │   ├── README.md
│   │   │   ├── data.tf
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── providers.tf
│   │   │   ├── variables.tf
│   │   │   └── versions.tf
│   │   └── state-management
│   │       ├── README.md
│   │       ├── main.tf
│   │       ├── providers.tf
│   │       ├── variables.tf
│   │       └── versions.tf
└── setup.cfg

How could pre-commit-terraform help solve your problem?

Reviewing this PR, it appears that this hook should work, but for some reason, I can't replicate

[MaxymVlasov]

What hook version do you use? In v1.64.0 it works fine

.pre-commit-config.yaml:

repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.64.0
  hooks:
    - id: terraform_validate
      args:
        - --init-args=-upgrade
        - --init-args=-get=true
        - --envs=AWS_DEFAULT_REGION="us-west-2"
        - --envs=AWS_ACCESS_KEY_ID="anaccesskey"
        - --envs=AWS_SECRET_ACCESS_KEY="asecretkey"

tree -d:

.
├── environment
│   ├── prd
│   └── qa
├── modules
│   ├── aws-acm-certificate
│   ├── aws-api-gateway
│   ├── aws-chatbot
│   ├── aws-dynamodb-table
│   ├── aws-ecr-repo
│   ├── aws-eks-cluster
│   ├── aws-eks-node-group
│   │   └── user_data
│   ├── aws-elasticache-cluster
│   ├── aws-environment
│   │   ├── templates
│   │   └── user_data
│   ├── aws-gitlab-ec2-manager
│   │   └── templates
│   ├── aws-grafana
│   ├── aws-iam-group
│   ├── aws-iam-role
│   ├── aws-iam-user
│   ├── aws-lambda-function
│   ├── aws-msk-cluster
│   ├── aws-rds-cluster
│   ├── aws-route53-zone
│   ├── aws-s3-bucket
│   ├── aws-sns-topic
│   ├── helm-release
│   ├── kube-manifest
│   ├── postgresql-env
│   └── rapp-services
│       ├── aws-core-api
│       ├── aws-iot-segway-max-plus
│       ├── aws-iot-segway-mock
│       └── aws-ops-web
├── path
├── services
│   ├── alerting
│   │   └── us-east-1
│   ├── chatbot
│   │   └── global
│   ├── dns
│   │   └── global
│   ├── docker-registry
│   │   └── global
│   ├── grafana
│   │   └── qa
│   ├── iam
│   │   └── global
│   ├── kafka
│   │   ├── prd
│   │   └── qa
│   ├── kubernetes
│   │   ├── prd
│   │   └── qa
│   ├── postgresql
│   │   └── qa
│   ├── rapp
│   │   ├── prd
│   │   │   └── p-1
│   │   │       ├── core-api
│   │   │       ├── iot-segway-max-plus
│   │   │       └── ops-web
│   │   └── qa
│   │       ├── commons
│   │       ├── qa-1
│   │       ├── qa-2
│   │       ├── qa-3
│   │       └── stg-1
│   ├── redis
│   │   ├── prd
│   │   └── qa
│   └── s3
│       ├── global
│       │   └── backend
│       └── qa
└── test

Feel free to reopen issue if will not works with latest version. And provide details specified in bug report template

[yermulnik]

Seems like I'm having sort of the same behavior as reporter (or I might be misinterpreting how this hook should work 🤔):

> tree broken/
broken/
├── test.tf
└── tf
    ├── code
    │   └── test.tf
    └── test.tf

2 directories, 3 files

> find broken/ -type f | while read F; do ls $F && cat $F && echo ====; done
broken/test.tf
test
====
broken/tf/test.tf
test
====
broken/tf/code/test.tf
test
====

> cat .pre-commit-config.yaml | egrep -v "^#"
repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  #rev: f3dd6deac411ee9144a22752db7a53923685fbce
  rev: v1.64.0
  hooks:
    - id: terraform_validate

> pre-commit clean
Cleaned /home/giermulnik/.cache/pre-commit.

> pre-commit run -a
[INFO] Initializing environment for https://github.com/antonbabenko/pre-commit-terraform.
Terraform validate.......................................................Passed

Obviously I'd expect terraform validate to error out on each of the files in nested dirs like this:

> cp broken/test.tf ./test.tf
> pre-commit run -a
Terraform validate.......................................................Failed
- hook id: terraform_validate
- exit code: 1

Validation failed: .
╷
│ Error: Argument or block definition required
│
│   on test.tf line 1:
│    1: test
│
│ An argument or block definition is required here. To set an argument, use
│ the equals sign "=" to introduce the argument value.
╵

[MaxymVlasov]

16:24 341 git:(main +)
➜ git status
On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   .pre-commit-config.yaml
        new file:   broken/test.tf
        new file:   broken/tf/code/test.tf
        new file:   broken/tf/test.tf

16:24 341 git:(main +)
➜ tree
.
└── broken
    ├── test.tf
    └── tf
        ├── code
        │   └── test.tf
        └── test.tf

3 directories, 3 files

16:24 341 git:(main +)
➜ cat .pre-commit-config.yaml
repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.64.0
  hooks:
    - id: terraform_validate
      args:
        - --init-args=-upgrade
        - --init-args=-get=true
        - --envs=AWS_DEFAULT_REGION="us-west-2"
        - --envs=AWS_ACCESS_KEY_ID="anaccesskey"
        - --envs=AWS_SECRET_ACCESS_KEY="asecretkey"

16:24 341 git:(main +)
➜ pre-commit run -a
Terraform validate.......................................................Failed
- hook id: terraform_validate
- exit code: 1

Init before validation failed: broken
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Argument or block definition required

  on test.tf line 1:
   1: fsdsdf

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.

Init before validation failed: broken/tf
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Argument or block definition required

  on test.tf line 1:
   1: fsdsdf

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.

Init before validation failed: broken/tf/code
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Argument or block definition required

  on test.tf line 1:
   1: fsdsdf

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.

$ find broken/ -type f | while read F; do ls $F && cat $F && echo ====; done
broken/tf/code/test.tf
fsdsdf
====
broken/tf/test.tf
fsdsdf
====
broken/test.tf
fsdsdf
====

[yermulnik]

Weird. This just doesn't behave like on your end:

> ls -la
total 16
drwx------ 2 giermulnik giermulnik  4096 Feb 16 16:50 .
drwx------ 6 giermulnik giermulnik 12288 Feb 16 16:44 ..

> git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint:   git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint:   git branch -m <name>
Initialised empty Git repository in /home/giermulnik/tmp/www/.git/

> cat <<EOF>.pre-commit-config.yaml
continue> repos:
continue> - repo: https://github.com/antonbabenko/pre-commit-terraform
continue>   rev: v1.64.0
continue>   hooks:
continue>     - id: terraform_validate
continue> EOF

> wget -q https://raw.githubusercontent.com/antonbabenko/pre-commit-terraform/master/.pre-commit-hooks.yaml

> mkdir -p broken/tf/code/

> for DIR in broken/tf/code broken/tf broken; do echo fsdsdf > $DIR/test.tf; done

> tree -a -I ".git"
.
├── broken
│   ├── test.tf
│   └── tf
│       ├── code
│       │   └── test.tf
│       └── test.tf
├── .pre-commit-config.yaml
└── .pre-commit-hooks.yaml

3 directories, 5 files

> pre-commit clean
Cleaned /home/giermulnik/.cache/pre-commit.

> pre-commit gc
0 repo(s) removed.

> pre-commit autoupdate
Updating https://github.com/antonbabenko/pre-commit-terraform ... already up to date.

> pre-commit run -a -v
[INFO] Initializing environment for https://github.com/antonbabenko/pre-commit-terraform.
Terraform validate...................................(no files to check)Skipped
- hook id: terraform_validate

[MaxymVlasov]

Did you run git add -A before pre-commit run -a?

[yermulnik]

Oh my god 🤦🏻 The most important step was missing from my setup 🤦🏻 Seems like topicstarter most probably has the same "issue".

[MaxymVlasov]

pre-commit can't check files that are not already added to git :)

[jamiezieziula]

That’s not the issue I’m having - my entire directory has been added to git. I’m testing this pre-commit on all files and intentionally making a change that should return a error from terraform validate.

On Wed, Feb 16, 2022 at 10:20 AM Maksym Vlasov @.***> wrote:

pre-commit can't check files that are not already added to git :)

— Reply to this email directly, view it on GitHub https://github.com/antonbabenko/pre-commit-terraform/issues/341#issuecomment-1041612019, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFCT74GELT5G7UMKCABQ56LU3O6CRANCNFSM5OCH76GQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

[yermulnik]

@jamiedick Please provide further info since what you provided initially didn't have any detailed info: show us git status of your repo, run terraform validate on "broken" file and show us result, cleanup pre-commit's cache (clean and gc), autoupdate pre-commit, show pre-commit config file from your repo, etc (you may get more insight looking at what Max (and I, though unintentionally wrong) did up above).