search

terraform-linters / tflint

A Pluggable Terraform Linter
Mozilla Public License 2.0
4.97k stars 357 forks source link

Recursive inspection in parallel #2021

Closed wata727 closed 6 months ago

wata727 commented 7 months ago

Fixes https://github.com/terraform-linters/tflint/issues/1943

As explained in #1943, this PR parallelizes recursive inspection by multi-processes.

The process that runs tflint --recursive is called the coordinator, and it starts a worker process for each directory. The worker process executes with --chdir in each directory and outputs issues in the dedicated JSON format, just like a normal TFLint command. The coordinator aggregates JSON output from each process via stdout, and provides feedback to the user.

flowchart TD

    User -- "tflint --recursive" --> Coordinator
    Coordinator -- "tflint --act-as-worker --chdir=subdir1" --> Worker1
    Coordinator -- "tflint --act-as-worker --chdir=subdir2" --> Worker2
    Coordinator -- "tflint --act-as-worker --chdir=subdir3" --> Worker3

The default parallelism is the number of CPUs. This can be controlled with the --max-workers flag, but should not need to be changed in most cases. In my tested repository (over 200 root modules with calls to over 40 child modules), 4 cores cut the time in half.

Before

$ time tflint --recursive
real    5m10.744s
user    5m53.837s
sys     2m36.451s

After

$ time tflint --recursive
real    2m24.805s
user    6m50.139s
sys     2m33.531s