search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.71k stars 9.55k forks source link

Terraform provider plugin cache not working #32936

Open a18-kaushik opened 1 year ago

a18-kaushik commented 1 year ago

Terraform Version

1.0.10

Terraform Configuration Files

plugin_cache_dir   = "C:/Users/a18.kaushik/AppData/Roaming/terraform.d/plugin-cache"

Debug Output

Initializing provider plugins...

Expected Behavior

terraform init execute successfully with cache provider

Actual Behavior

Error: Failed to install provider │ │ Error while installing hashicorp/aws v4.60.0: failed to create directory │ C:\Users\a18.kaushik\Desktop.terraform\providers\registry.terraform.io\hashicorp\aws\4.60.0\windows_amd64: │ mkdir │ C:\Users\a18.kaushik\Desktop.terraform\providers\registry.terraform.io\hashicorp\aws\4.60.0\windows_amd64: │ Cannot create a file when that file already exists.

Steps to Reproduce

  1. make terraform.rc file with above configuration file settings
  2. create "plugin-cache" folder in "C:\Users\a18.kaushik\AppData\Roaming\terraform.d"
  3. run terraform init

Additional Context

followed documentation for provider plugin cache : https://developer.hashicorp.com/terraform/cli/config/config-file#provider-installation

References

No response

jbardin commented 1 year ago

Hi @a18-kaushik,

Can you please verify that the behavior is the same on a current release of Terraform? Some behaviors have changed and bugs have been fixed since the v1.0 releases.

Thanks!

a18-kaushik commented 1 year ago

Hi @jbardin i tried with version 1.4.2 but the problem persists

debug logs:

Initializing provider plugins...

a18-kaushik commented 1 year ago

@jbardin did you get a chance to look into this? thanks

yongzhang commented 1 year ago

I upgraded from v1.3.x to v1.4.x, looks like install from caching no longer works, or at least no longer works as expected. I have multiple stack folders using the same providers, terraform init installs all of the providers in the first stack folder, but it still downloads the same in the following folders not from cache.

jbardin commented 1 year ago

@a18-kaushik, The error is a Windows error from linking the provider files. It will take some time to reproduce and investigate what exactly is failing on that platform.

a18-kaushik commented 1 year ago

@jbardin okay noted!

phyber commented 1 year ago

I believe I have this issue on Linux (amd64) too with Terraform 1.4.6. Terraform will scan my plugin-cache, find appropriate versions of the (in this case) AWS provider, and proceed to re-download it anyway. There seem to be no logs in the TF_LOG=TRACE that say why it decided to redownload the provider.

It may also be re-downloading other providers constantly too, but since they're much smaller it's harder to notice.

The plugin cache used to work correctly, I'm unsure when it broke, but it's definitely been months for me at this point.

fyqtian commented 1 year ago

version 1.5.2 on Mac(amd64) is same behavior `Initializing provider plugins...

fyqtian commented 1 year ago

I believe that if the .terraform.lock.hcl file is absent, local caching will not be utilized.

https://github.com/hashicorp/terraform/blob/v1.4.6/internal/providercache/installer.go#L387-L398

magodo commented 1 year ago

Same thing happens to me (on Linux), where I have to use plugin_cache_may_break_dependency_lock_file = true to make the cache to work. This isn't an issue back in the v1.3.x.

tnc-tschmidt commented 1 year ago

Same. Essentially if the there is no lock file it will always download from upstream and not respect the plugin_cache. For me setting plugin_cache_may_break_dependency_lock_file does not help and it still redownloads it.

My unfortunate work around is to pin the aws provider version and copy in a .terraform.lock.hcl file with the hashes. Whicht seems like this is expected behavior but I don't understand the reasoning behind this decision. 

                // An existing cache entry is only an acceptable choice
                // if there is already a lock file entry for this provider
                // and the cache entry matches its checksums.
                //
                // If there was no lock file entry at all then we need to
                // install the package for real so that we can lock as complete
                // as possible a set of checksums for all of this provider's
                // packages.
jugger27 commented 11 months ago

Same problem here with MacOS ARM64 and TF 1.5.7. Rolled back to using TF 1.3.10.

alxndr13 commented 8 months ago

Same thing happens to me (on Linux), where I have to use plugin_cache_may_break_dependency_lock_file = true to make the cache to work. This isn't an issue back in the v1.3.x.

can confirm. Only accesses the local plugin cache if I set the environment variable.

Terraform v1.7.3
on linux_amd64

but: this may be okay if I understood the docs correctly.

By default Terraform will use packages from the global cache directory only if they match at least one of the checksums recorded in the dependency lock file for that provider.

in my case, I regenerate the dependency lock file each time. (as the terraform code gets generated by some other tooling) This is why I have to set that flag to make things work.