crw
commented
1 year ago Thanks for this request! It may be a duplicate of https://github.com/hashicorp/terraform/issues/32901.
Open BrianTuduri opened 1 year ago
crw
commented
1 year ago Thanks for this request! It may be a duplicate of https://github.com/hashicorp/terraform/issues/32901.
apparentlymart
commented
1 year ago Hi @BrianTuduri! Sorry for this strange behavior and thanks for reporting it.
I've been trying to make sense of the series of messages in the trace log you shared, but there are some oddities that I can't explain and I wonder if you have some additional context that might help me understand what's going on.
The main thing that confuses me is that I see log lines about installing provider packages from the cache before Terraform actually reached the point of "Initializing provider plugins..." and thus before it should know which versions it's supposed to be using:
2023-07-10T13:15:05.532Z [TRACE] providercache.Dir.InstallPackage: installing registry.terraform.io/hashicorp/helm v2.10.1 from https://releases.hashicorp.com/terraform-provider-helm/2.10.1/terraform-provider-helm_2.10.1_linux_amd64.zip
2023-07-10T13:15:17.124Z [TRACE] providercache.Dir.LinkFromOtherCache: linking registry.terraform.io/hashicorp/helm v2.10.1 from existing cache /tf_cache/registry.terraform.io/hashicorp/helm/2.10.1/linux_amd64 to .terraform/providers/registry.terraform.io/hashicorp/helm/2.10.1/linux_amd64
- Installed hashicorp/helm v2.10.1 (signed by HashiCorp)
2023-07-10T13:15:05.121Z [TRACE] providercache.Dir.InstallPackage: installing registry.terraform.io/rancher/rancher2 v3.0.2 from https://releases.hashicorp.com/terraform-provider-rancher2/3.0.2/terraform-provider-rancher2_3.0.2_linux_amd64.zip
2023-07-10T13:15:05.122Z [TRACE] providercache.Dir.LinkFromOtherCache: linking registry.terraform.io/rancher/rancher2 v3.0.2 from existing cache /tf_cache/registry.terraform.io/rancher/rancher2/3.0.2/linux_amd64 to .terraform/providers/registry.terraform.io/rancher/rancher2/3.0.2/linux_amd64
- Installed rancher/rancher2 v3.0.2 (signed by a HashiCorp partner, key ID 2EEB0F9AD44A135C)
Initializing provider plugins...
- Finding latest version of hashicorp/kubernetes...
2023-07-10T13:14:43.550Z [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
[etc]However, I notice that the timestamp on the final log line is 13:14:43.55 while the earlier one was 13:15:05.532, so it seems like perhaps these log lines have somehow been misordered by the system that you used to collect them?
If I try to slot those lines into the place where their timestamps suggest they should go then the sequence of events does seem plausible, so if I assume that this was just a log aggregation quirk then indeed it does seem like Terraform is somehow concluding that the items in the cache are not valid, but unfortunately the logs here don't seem to give any indication as to why, so I think this might take some additional debugging.
I see in your CLI configuration you enabled plugin_cache_may_break_dependency_lock_file = true, and the code which handles that special exception is written to generate a log line acknowledging it:
[WARN] plugin_cache_may_break_dependency_lock_file: Using global cache dir package for %s v%s even though it doesn't match this configuration's dependency lock fileThere seems to be no such message in the logs you shared, which (assuming that there are no log lines missing from what you shared) suggests that the plugin_cache_may_break_dependency_lock_file was not active for some reason.
The log lines we can see seem consistent with how Terraform would behave if that setting were not in effect and if the cache directory checksums were unverifyable, so it seems like Terraform is for some reason not honoring that setting, as opposed to the relevant log lines just being absent in your trace log.
So of course this raises the question of why exactly Terraform cannot "see" the plugin_cache_may_break_dependency_lock_file setting.
Although I don't yet have a direct explanation for why this would be true, I have a hypothesis that for some reason that setting might be ineffective when the CLI configuration location is overridden using the TF_CLI_CONFIG_FILE environment variable. Of course that's not the intended behavior, but setting that environment variable does alter some of the configuration loading behavior (since it's loading exactly one file rather than scanning for possibly many files in different directories), and so there might be a bug in how that different situation gets handled.
Is it possible for you to temporarily arrange for that CLI configuration file to be in one of the standard search locations, so you can see if the incorrect behavior persists when you don't set TF_CLI_CONFIG_FILE? Thanks!
Surya4DevOps
commented
1 year ago Even I faced similar issue and I spent lot of time in fixing this issue by trying different possibilities. Finally here is what I did as workaround for the issue in terraform version v1.5.3.
First make sure all the downloaded plugins from terraform repo are available at $HOME/.terraform.d/tf-cache
Add below .terraformrc config file to home directory.
cat <<EOF > $HOME/.terraformrc
provider_installation {
filesystem_mirror {
path="${HOME}/.terraform.d/tf-cache/"
include=["registry.terraform.io/hashicorp/*"]
}
direct {
exclude=["registry.terraform.io/hashicorp/*"]
}
}
plugin_cache_dir="${HOME}/.terraform.d/tf-cache/"
disable_checkpoint=true
EOFWhile running terraform init use --plugin-dir option, it started working.
terraform init -backend-config=terraform-state.tfvars --plugin-dir "$HOME/.terraform.d/tf-cache/"
Hope this helps.
magodo
commented
7 months ago I have a similar issue with a fresh new environment setup to run terraform. I have following config:
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"Everytime when I run terraform init, it downloads the provider again even though they exist in the cache.
The log looks like below:
2024-03-07T13:38:34.267+0800 [INFO] Terraform version: 1.7.4
2024-03-07T13:38:34.267+0800 [DEBUG] using github.com/hashicorp/go-tfe v1.41.0
2024-03-07T13:38:34.267+0800 [DEBUG] using github.com/hashicorp/hcl/v2 v2.19.1
2024-03-07T13:38:34.267+0800 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2024-03-07T13:38:34.267+0800 [DEBUG] using github.com/zclconf/go-cty v1.14.1
2024-03-07T13:38:34.267+0800 [INFO] Go runtime version: go1.21.5
2024-03-07T13:38:34.267+0800 [INFO] CLI args: []string{"terraform", "init"}
2024-03-07T13:38:34.267+0800 [TRACE] Stdout is a terminal of width 198
2024-03-07T13:38:34.267+0800 [TRACE] Stderr is a terminal of width 198
2024-03-07T13:38:34.267+0800 [TRACE] Stdin is a terminal
2024-03-07T13:38:34.267+0800 [DEBUG] Attempting to open CLI config file: /home/magodo/.terraformrc
2024-03-07T13:38:34.267+0800 [INFO] Loading CLI configuration from /home/magodo/.terraformrc
2024-03-07T13:38:34.267+0800 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2024-03-07T13:38:34.267+0800 [DEBUG] ignoring non-existing provider search directory /home/magodo/.terraform.d/plugins
2024-03-07T13:38:34.267+0800 [DEBUG] ignoring non-existing provider search directory /home/magodo/.local/share/terraform/plugins
2024-03-07T13:38:34.267+0800 [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2024-03-07T13:38:34.267+0800 [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2024-03-07T13:38:34.268+0800 [INFO] CLI command args: []string{"init"}
Initializing the backend...
2024-03-07T13:38:34.269+0800 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2024-03-07T13:38:34.269+0800 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2024-03-07T13:38:34.269+0800 [DEBUG] New state was assigned lineage "faa16da8-6f83-086e-e859-5bb56fd36ed8"
2024-03-07T13:38:34.269+0800 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2024-03-07T13:38:34.269+0800 [TRACE] Meta.Backend: instantiated backend of type
2024-03-07T13:38:34.269+0800 [DEBUG] checking for provisioner in "."
2024-03-07T13:38:34.270+0800 [DEBUG] checking for provisioner in "/usr/bin"
2024-03-07T13:38:34.270+0800 [TRACE] Meta.Backend: backend does not support operations, so wrapping it in a local backend
2024-03-07T13:38:34.270+0800 [TRACE] backend/local: state manager for workspace "default" will:
- read initial snapshot from terraform.tfstate
- write new snapshots to terraform.tfstate
- create any backup at terraform.tfstate.backup
2024-03-07T13:38:34.270+0800 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2024-03-07T13:38:34.270+0800 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2024-03-07T13:38:34.270+0800 [TRACE] statemgr.Filesystem: read nil snapshot
Initializing provider plugins...
- Finding latest version of hashicorp/azurerm...
2024-03-07T13:38:34.270+0800 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2024-03-07T13:38:34.270+0800 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2024-03-07T13:38:34.564+0800 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/azurerm/versions
2024-03-07T13:38:34.564+0800 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/azurerm/versions
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2024-03-07T13:38:34.837+0800 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform: no such file or directory
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: scanning directory /home/magodo/.terraform.d/plugin-cache
2024-03-07T13:38:34.837+0800 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/azuread v2.38.0 for linux_amd64 at /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/azuread/2.38.0/linux_amd64
2024-03-07T13:38:34.837+0800 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/azurerm v3.94.0 for linux_amd64 at /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/azurerm/3.94.0/linux_amd64
2024-03-07T13:38:34.837+0800 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/time v0.9.1 for linux_amd64 at /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/time/0.9.1/linux_amd64
2024-03-07T13:38:34.837+0800 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/tls v4.0.4 for linux_amd64 at /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/tls/4.0.4/linux_amd64
2024-03-07T13:38:34.837+0800 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/tls v4.0.5 for linux_amd64 at /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/tls/4.0.5/linux_amd64
2024-03-07T13:38:34.837+0800 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/vancluever/acme v2.20.2 for linux_amd64 at /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/vancluever/acme/2.20.2/linux_amd64
2024-03-07T13:38:34.837+0800 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/vancluever/acme v2.7.1 for linux_amd64 at /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/vancluever/acme/2.7.1/linux_amd64
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: including /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/azuread/2.38.0/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/azuread 2.38.0
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: including /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/azurerm/3.94.0/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/azurerm 3.94.0
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: including /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/time/0.9.1/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/time 0.9.1
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: including /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/tls/4.0.4/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/tls 4.0.4
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: including /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/tls/4.0.5/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/tls 4.0.5
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: including /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/vancluever/acme/2.7.1/linux_amd64 as a candidate package for registry.terraform.io/vancluever/acme 2.7.1
2024-03-07T13:38:34.837+0800 [TRACE] providercache.fillMetaCache: including /home/magodo/.terraform.d/plugin-cache/registry.terraform.io/vancluever/acme/2.20.2/linux_amd64 as a candidate package for registry.terraform.io/vancluever/acme 2.20.2
2024-03-07T13:38:34.837+0800 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/azurerm/3.94.0/download/linux/amd64
2024-03-07T13:38:34.837+0800 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/azurerm/3.94.0/download/linux/amd64
2024-03-07T13:38:35.136+0800 [DEBUG] GET https://releases.hashicorp.com/terraform-provider-azurerm/3.94.0/terraform-provider-azurerm_3.94.0_SHA256SUMS
2024-03-07T13:38:35.136+0800 [TRACE] HTTP client GET request to https://releases.hashicorp.com/terraform-provider-azurerm/3.94.0/terraform-provider-azurerm_3.94.0_SHA256SUMS
2024-03-07T13:38:35.544+0800 [DEBUG] GET https://releases.hashicorp.com/terraform-provider-azurerm/3.94.0/terraform-provider-azurerm_3.94.0_SHA256SUMS.72D7468F.sig
2024-03-07T13:38:35.545+0800 [TRACE] HTTP client GET request to https://releases.hashicorp.com/terraform-provider-azurerm/3.94.0/terraform-provider-azurerm_3.94.0_SHA256SUMS.72D7468F.sig
- Installing hashicorp/azurerm v3.94.0...
2024-03-07T13:38:35.636+0800 [TRACE] providercache.Dir.InstallPackage: installing registry.terraform.io/hashicorp/azurerm v3.94.0 from https://releases.hashicorp.com/terraform-provider-azurerm/3.94.0/terraform-provider-azurerm_3.94.0_linux_amd64.zip
2024-03-07T13:38:35.636+0800 [TRACE] HTTP client GET request to https://releases.hashicorp.com/terraform-provider-azurerm/3.94.0/terraform-provider-azurerm_3.94.0_linux_amd64.zip
Only when I enable plugin_cache_may_break_dependency_lock_file, it stopps the re-downloading. I suspect this is also unexpected?
haidaraM
commented
4 days ago Same problem here. If you enable plugin_cache_may_break_dependency_lock_file, you now get a warning which a bit annoying. This used to work before, IIRC.
Terraform Version
Terraform Configuration Files
Debug Output
Expected Behavior
I was expecting Terraform to use cached vendor plugins instead of downloading them again.
Actual Behavior
Terraform is downloading again the plugins from the providers, even though they are already present in the cache and apparently it detects them, if I am making a mistake please let me know.
Steps to Reproduce
terraform initterraform applyAdditional Context
I am using Jenkins 2.235.5 and I have configured an agent template with a Terraform image. This image already includes a directory and volume "tf_cache", as well as the environment variables: TF_PLUGIN_CACHE_DIR="/tf_cache" and ENV TF_CLI_CONFIG_FILE="/tf_cache/.terraformrc". Additionally, I copy a file named .terraformrc to "tf_cache" which contains the following settings:
These settings are intended to set the plugin cache directory of the provider plugins to /tf_cache. This setting works as expected up to Terraform version 1.3.9. However, starting with version 1.4.0, Terraform re-downloads vendor plugins, even though they are already present in the cache.
The "tf_cache" volume is mounted on the server where Jenkins is running. Although versions after 1.4.0 of Terraform persist the providers in "tf_cache", they do not use them and instead redownload them. This results in inefficient and undesirable behavior.
References
I am not aware of any other problems or pull requests that should be linked here.