search

gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.
https://terragrunt.gruntwork.io/
MIT License
8.12k stars 986 forks source link

Parallelism not disabled on `init` when terraform plugin cache is used #3476

Open loganmzz opened 1 month ago

loganmzz commented 1 month ago

Describe the bug

Document states:

The exception is the terraform init command, which is always executed sequentially if the terraform plugin cache is used.

However, I enabled plugin cache with environment variable TF_PLUGIN_CACHE_DIR and parallelism still apply (whatever it is specified or not).

Steps To Reproduce

Create stack with several Terragrunt modules and run:

export TF_PLUGIN_CACHE_DIR="$(pwd)/.tf-plugin-cache" &&
mkdir -p "${TF_PLUGIN_CACHE_DIR}" &&
terragrunt run-all init -reconfigure -upgrade &&
true

Expected behavior

Successful Terraform initialization of all Terragrunt modules without the famous:

  │ Error: Required plugins are not installed
  │ 
  │ The installed provider plugins are not consistent with the packages
  │ selected in the dependency lock file:
  │   - registry.terraform.io/hashicorp/aws: the cached package for registry.terraform.io/hashicorp/aws 5.72.0 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file
  │ 
  │ Terraform uses external plugins to integrate with a variety of different
  │ infrastructure services. To download the plugins required for this
  │ configuration, run:
  │   terraform init

Nice to haves

09:02:19.776 STDOUT [0001] terraform: Initializing the backend...
09:02:19.776 STDOUT [0002] terraform: Initializing the backend...
09:02:19.813 STDOUT [0003] terraform: Initializing the backend...
09:02:19.814 STDOUT [0004] terraform: Initializing the backend...
09:02:22.112 STDOUT [0004] terraform: 
09:02:22.112 STDOUT [0004] terraform: Successfully configured the backend "s3"! Terraform will automatically
09:02:22.112 STDOUT [0004] terraform: use this backend unless the backend configuration changes.
09:02:22.364 STDOUT [0002] terraform: 
09:02:22.364 STDOUT [0002] terraform: Successfully configured the backend "s3"! Terraform will automatically
09:02:22.364 STDOUT [0002] terraform: use this backend unless the backend configuration changes.
09:02:22.469 STDOUT [0001] terraform: 
09:02:22.469 STDOUT [0001] terraform: Successfully configured the backend "s3"! Terraform will automatically
09:02:22.469 STDOUT [0001] terraform: use this backend unless the backend configuration changes.
09:02:22.652 STDOUT [0003] terraform: 
09:02:22.652 STDOUT [0003] terraform: Successfully configured the backend "s3"! Terraform will automatically
09:02:22.652 STDOUT [0003] terraform: use this backend unless the backend configuration changes.
09:02:24.067 STDOUT [0004] terraform: Initializing provider plugins...
09:02:24.067 STDOUT [0004] terraform: - Finding latest version of hashicorp/aws...
09:02:24.519 STDOUT [0004] terraform: - Installing hashicorp/aws v5.72.0...
09:02:24.836 STDOUT [0002] terraform: Initializing provider plugins...
09:02:24.836 STDOUT [0002] terraform: - terraform.io/builtin/terraform is built in to Terraform
09:02:24.836 STDOUT [0002] terraform: - Finding hashicorp/kubernetes versions matching "2.24.0"...
09:02:25.261 STDOUT [0001] terraform: Initializing provider plugins...
09:02:25.261 STDOUT [0001] terraform: - terraform.io/builtin/terraform is built in to Terraform
09:02:25.261 STDOUT [0001] terraform: - Finding latest version of hashicorp/aws...
09:02:25.549 STDOUT [0001] terraform: - Finding alekc/kubectl versions matching "~> 2.0"...
09:02:25.678 STDOUT [0001] terraform: - Finding hashicorp/kubernetes versions matching "2.23.0"...
09:02:25.853 STDOUT [0003] terraform: Initializing provider plugins...
09:02:25.853 STDOUT [0003] terraform: - terraform.io/builtin/terraform is built in to Terraform
09:02:25.853 STDOUT [0003] terraform: - Finding hashicorp/helm versions matching "2.12.1"...
09:02:25.986 STDOUT [0001] terraform: - Finding hashicorp/helm versions matching "2.11.0"...
09:02:26.160 STDOUT [0003] terraform: - Finding alekc/kubectl versions matching "~> 2.0"...
09:02:26.286 STDOUT [0003] terraform: - Finding hashicorp/kubernetes versions matching "2.24.0"...
09:02:28.041 STDOUT [0002] terraform: - Finding hashicorp/helm versions matching "2.12.1"...
09:02:31.415 STDOUT [0003] terraform: - Finding latest version of hashicorp/aws...
09:02:31.939 STDOUT [0003] terraform: - Using previously-installed hashicorp/helm v2.12.1
09:02:31.943 STDOUT [0001] terraform: - Installing hashicorp/aws v5.72.0...
09:02:32.073 STDOUT [0003] terraform: - Using previously-installed alekc/kubectl v2.1.0
09:02:32.243 STDOUT [0003] terraform: - Using previously-installed hashicorp/kubernetes v2.24.0
09:02:33.163 STDOUT [0002] terraform: - Finding latest version of hashicorp/aws...
09:02:33.415 STDOUT [0002] terraform: - Using previously-installed hashicorp/kubernetes v2.24.0
09:02:33.551 STDOUT [0002] terraform: - Using previously-installed hashicorp/helm v2.12.1
09:02:33.827 STDOUT [0002] terraform: - Installing hashicorp/aws v5.72.0...
09:02:37.197 STDOUT [0004] terraform: - Installed hashicorp/aws v5.72.0 (signed by HashiCorp)
09:02:37.198 STDOUT [0004] terraform: Terraform has made some changes to the provider dependency selections recorded
09:02:37.198 STDOUT [0004] terraform: in the .terraform.lock.hcl file. Review those changes and commit them to your
09:02:37.198 STDOUT [0004] terraform: version control system if they represent changes you intended to make.
09:02:37.198 STDOUT [0004] terraform: Terraform has been successfully initialized!
09:02:37.198 STDOUT [0004] terraform: 
09:02:37.198 STDOUT [0004] terraform: You may now begin working with Terraform. Try running "terraform plan" to see
09:02:37.198 STDOUT [0004] terraform: any changes that are required for your infrastructure. All Terraform commands
09:02:37.198 STDOUT [0004] terraform: should now work.
09:02:37.198 STDOUT [0004] terraform: If you ever set or change modules or backend configuration for Terraform,
09:02:37.198 STDOUT [0004] terraform: rerun this command to reinitialize your working directory. If you forget, other
09:02:37.198 STDOUT [0004] terraform: commands will detect it and remind you to do so if necessary.
09:02:37.527 STDOUT [0003] terraform: - Installing hashicorp/aws v5.72.0...
09:03:02.970 STDOUT [0002] terraform: - Installed hashicorp/aws v5.72.0 (signed by HashiCorp)
09:03:02.970 STDOUT [0002] terraform: Terraform has made some changes to the provider dependency selections recorded
09:03:02.970 STDOUT [0002] terraform: in the .terraform.lock.hcl file. Review those changes and commit them to your
09:03:02.970 STDOUT [0002] terraform: version control system if they represent changes you intended to make.
09:03:02.971 STDOUT [0002] terraform: Terraform has been successfully initialized!
09:03:02.971 STDOUT [0002] terraform: 
09:03:02.971 STDOUT [0002] terraform: You may now begin working with Terraform. Try running "terraform plan" to see
09:03:02.971 STDOUT [0002] terraform: any changes that are required for your infrastructure. All Terraform commands
09:03:02.971 STDOUT [0002] terraform: should now work.
09:03:02.971 STDOUT [0002] terraform: If you ever set or change modules or backend configuration for Terraform,
09:03:02.971 STDOUT [0002] terraform: rerun this command to reinitialize your working directory. If you forget, other
09:03:02.971 STDOUT [0002] terraform: commands will detect it and remind you to do so if necessary.
09:03:08.418 STDOUT [0001] terraform: - Installed hashicorp/aws v5.72.0 (signed by HashiCorp)
09:03:08.571 STDOUT [0001] terraform: - Using previously-installed alekc/kubectl v2.1.0
09:03:08.700 STDOUT [0001] terraform: - Using previously-installed hashicorp/kubernetes v2.23.0
09:03:08.880 STDOUT [0001] terraform: - Using previously-installed hashicorp/helm v2.11.0
09:03:08.880 STDOUT [0001] terraform: Terraform has made some changes to the provider dependency selections recorded
09:03:08.880 STDOUT [0001] terraform: in the .terraform.lock.hcl file. Review those changes and commit them to your
09:03:08.880 STDOUT [0001] terraform: version control system if they represent changes you intended to make.
09:03:08.880 STDOUT [0001] terraform: Terraform has been successfully initialized!
09:03:08.880 STDOUT [0001] terraform: 
09:03:08.880 STDOUT [0001] terraform: You may now begin working with Terraform. Try running "terraform plan" to see
09:03:08.881 STDOUT [0001] terraform: any changes that are required for your infrastructure. All Terraform commands
09:03:08.881 STDOUT [0001] terraform: should now work.
09:03:08.881 STDOUT [0001] terraform: If you ever set or change modules or backend configuration for Terraform,
09:03:08.881 STDOUT [0001] terraform: rerun this command to reinitialize your working directory. If you forget, other
09:03:08.881 STDOUT [0001] terraform: commands will detect it and remind you to do so if necessary.

Versions

Additional context

N/A

yhakbar commented 1 month ago

Hey @loganmzz , do you mind submitting a pull request to update the documentation that you found saying that?

The Provider Cache Server is what we recommend users use for concurrent access to provider cache across units (terragrunt.hcl files).