search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.66k stars 9.55k forks source link

Option to upgrade to latest available provider versions as part of "terraform providers lock" #27388

Open thomasboussekey opened 3 years ago

thomasboussekey commented 3 years ago

Terraform Version

0.14.3

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket = "terraform-remote-state"
    region = "eu-west-1"
  }

  required_version = "~> 0.14"
  required_providers {
    postgresql = {
      source  = "cyrilgdn/postgresql"
      version = "~> 1.9.0"
    }
    google = {
      source  = "hashicorp/google"
      version = "~> 3.22"
    }
    vault = {
      source  = "hashicorp/vault"
      version = "~> 2.17"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.0"
    }
  }
}

Debug Output

None

Crash Output

None

Expected Behavior

Want to update terraform providers to a fresher version as provided in the configuration.

Actual Behavior

Unable to change the version of the providers I found a workaround: deleting and re-creating the provider dependency lock file

Steps to Reproduce

  1. Update the `backend.tf context file with new versions
  2. Run the following make recipe to perform a terraform init -upgrade to upgrade providers for Linux & Darwin
Provider Version in lock file expected new one
hashicorp/google 3.4.0 ~> 3.22
hashicorp/vault 2.7.1 ~> 2.17
hashicorp/random 2.2.1 ~> 3.0

Make recipe:

refresh_providers:
    $(TERRAFORM) init -upgrade
    # In failure on first attempt, perform a single-platform action:
    # $(TERRAFORM) providers lock -platform=linux_amd64
    $(TERRAFORM) providers lock -platform=darwin_amd64 -platform=linux_amd64

Note: ${TERRAFORM} is wrapper calling a shell script ../../../../../commons/bin/terraform.sh that starts TERRAORM.

I have the following output:

$ make refresh_providers
../../../../../commons/bin/terraform.sh init -upgrade
Reading required version from terraform file, constraint: ~> 0.14
Matched version: 0.14.3
Switched terraform to version "0.14.3" 
Upgrading modules...
- connect_databases in modules/databases
- connect_extensions in modules/extensions

Initializing the backend...

Initializing provider plugins...
- Finding cyrilgdn/postgresql versions matching "~> 1.9.0"...
- Finding hashicorp/google versions matching "~> 3.22"...
- Finding hashicorp/vault versions matching "~> 2.17"...
- Finding hashicorp/random versions matching "~> 3.0"...
- Using cyrilgdn/postgresql v1.9.0 from the shared cache directory
- Using hashicorp/google v3.51.0 from the shared cache directory
- Using hashicorp/vault v2.17.0 from the shared cache directory
- Using hashicorp/random v3.0.0 from the shared cache directory

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
# In failure on first attempt, perform a single-platform action:
# ../../../../../commons/bin/terraform.sh providers lock -platform=linux_amd64
../../../../../commons/bin/terraform.sh providers lock -platform=darwin_amd64 -platform=linux_amd64
Reading required version from terraform file, constraint: ~> 0.14
Matched version: 0.14.3
Switched terraform to version "0.14.3" 
- Fetching cyrilgdn/postgresql 1.9.0 for darwin_amd64...
- Obtained cyrilgdn/postgresql checksums for darwin_amd64 (self-signed, key ID 3918DD444A3876A6)
- Fetching hashicorp/google 3.51.0 for darwin_amd64...
- Fetching hashicorp/vault 2.17.0 for darwin_amd64...
- Fetching hashicorp/random 3.0.0 for darwin_amd64...

Error: Could not retrieve providers for locking

Terraform failed to fetch the requested providers for darwin_amd64 in order to
calculate their checksums: some providers could not be installed:
- registry.terraform.io/hashicorp/google: the current package for
registry.terraform.io/hashicorp/google 3.51.0 doesn't match any of the
checksums previously recorded in the dependency lock file
- registry.terraform.io/hashicorp/random: the current package for
registry.terraform.io/hashicorp/random 3.0.0 doesn't match any of the
checksums previously recorded in the dependency lock file
- registry.terraform.io/hashicorp/vault: the current package for
registry.terraform.io/hashicorp/vault 2.17.0 doesn't match any of the
checksums previously recorded in the dependency lock file.

make: *** [refresh_providers] Error 1

Workaround to perform the provider upgrade

Fix step 0,Remove the lock file

$ mv .terraform.lock.hcl .terraform.lock.hcl.old

Fix step 1, using a single platform provider

refresh_providers:
    $(TERRAFORM) init -upgrade
    # In failure on first attempt, perform a single-platform action:
    $(TERRAFORM) providers lock -platform=linux_amd64
    # $(TERRAFORM) providers lock -platform=darwin_amd64 -platform=linux_amd64
$ make refresh_providers
../../../../../commons/bin/terraform.sh init -upgrade
Reading required version from terraform file, constraint: ~> 0.14
Matched version: 0.14.3
Switched terraform to version "0.14.3" 
Upgrading modules...
- connect_databases in modules/databases
- connect_extensions in modules/extensions

Initializing the backend...

Initializing provider plugins...
- Finding cyrilgdn/postgresql versions matching "~> 1.9.0"...
- Finding hashicorp/google versions matching "~> 3.22"...
- Finding hashicorp/vault versions matching "~> 2.17"...
- Finding hashicorp/random versions matching "~> 3.0"...
- Using hashicorp/google v3.51.0 from the shared cache directory
- Using hashicorp/vault v2.17.0 from the shared cache directory
- Using hashicorp/random v3.0.0 from the shared cache directory
- Using cyrilgdn/postgresql v1.9.0 from the shared cache directory

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
# In failure on first attempt, perform a single-platform action:
../../../../../commons/bin/terraform.sh providers lock -platform=linux_amd64
Reading required version from terraform file, constraint: ~> 0.14
Matched version: 0.14.3
Switched terraform to version "0.14.3" 
- Fetching cyrilgdn/postgresql 1.9.0 for linux_amd64...
- Obtained cyrilgdn/postgresql checksums for linux_amd64 (self-signed, key ID 3918DD444A3876A6)
- Fetching hashicorp/google 3.51.0 for linux_amd64...
- Obtained hashicorp/google checksums for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/vault 2.17.0 for linux_amd64...
- Obtained hashicorp/vault checksums for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/random 3.0.0 for linux_amd64...
- Obtained hashicorp/random checksums for linux_amd64 (signed by HashiCorp)

Success! Terraform has updated the lock file.

Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

# ../../../../../commons/bin/terraform.sh providers lock -platform=darwin_amd64 -platform=linux_amd64

Fix step 2, Add the second platform

refresh_providers:
    $(TERRAFORM) init -upgrade
    # In failure on first attempt, perform a single-platform action:
    # $(TERRAFORM) providers lock -platform=linux_amd64
    $(TERRAFORM) providers lock -platform=darwin_amd64 -platform=linux_amd64
$ make refresh_providers
../../../../../commons/bin/terraform.sh init -upgrade
Reading required version from terraform file, constraint: ~> 0.14
Matched version: 0.14.3
Switched terraform to version "0.14.3" 
Upgrading modules...
- connect_databases in modules/databases
- connect_extensions in modules/extensions

Initializing the backend...

Initializing provider plugins...
- Finding cyrilgdn/postgresql versions matching "~> 1.9.0"...
- Finding hashicorp/google versions matching "~> 3.22"...
- Finding hashicorp/vault versions matching "~> 2.17"...
- Finding hashicorp/random versions matching "~> 3.0"...
- Using cyrilgdn/postgresql v1.9.0 from the shared cache directory
- Using hashicorp/google v3.51.0 from the shared cache directory
- Using hashicorp/vault v2.17.0 from the shared cache directory
- Using hashicorp/random v3.0.0 from the shared cache directory

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
# In failure on first attempt, perform a single-platform action:
# ../../../../../commons/bin/terraform.sh providers lock -platform=linux_amd64
../../../../../commons/bin/terraform.sh providers lock -platform=darwin_amd64 -platform=linux_amd64
Reading required version from terraform file, constraint: ~> 0.14
Matched version: 0.14.3
Switched terraform to version "0.14.3" 
- Fetching hashicorp/random 3.0.0 for darwin_amd64...
- Obtained hashicorp/random checksums for darwin_amd64 (signed by HashiCorp)
- Fetching cyrilgdn/postgresql 1.9.0 for darwin_amd64...
- Obtained cyrilgdn/postgresql checksums for darwin_amd64 (self-signed, key ID 3918DD444A3876A6)
- Fetching hashicorp/google 3.51.0 for darwin_amd64...
- Obtained hashicorp/google checksums for darwin_amd64 (signed by HashiCorp)
- Fetching hashicorp/vault 2.17.0 for darwin_amd64...
- Obtained hashicorp/vault checksums for darwin_amd64 (signed by HashiCorp)
- Fetching cyrilgdn/postgresql 1.9.0 for linux_amd64...
- Obtained cyrilgdn/postgresql checksums for linux_amd64 (self-signed, key ID 3918DD444A3876A6)
- Fetching hashicorp/google 3.51.0 for linux_amd64...
- Obtained hashicorp/google checksums for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/vault 2.17.0 for linux_amd64...
- Obtained hashicorp/vault checksums for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/random 3.0.0 for linux_amd64...
- Obtained hashicorp/random checksums for linux_amd64 (signed by HashiCorp)

Success! Terraform has updated the lock file.

Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

Fix step 3, Remove the backup terraform lock file

rm .terraform.lock.hcl.old

Additional Context

References

pselle commented 3 years ago

Hi @thomasboussekey! You describe your issue as Want to update terraform providers to a fresher version as provided in the configuration. but haven't outlined what simplified steps (what config? what terraform commands specifically?) to recreate the issue you saw (although hopefully your detailed workaround description might help a future reader).

Could you try to describe the issue you're seeing (the bug in Terraform you are describing) with a simplified configuration so we might be able to reproduce the issue?

thomasboussekey commented 3 years ago

Hi @pselle, Thanks for your comment I put my answers below

Hi @thomasboussekey! You describe your issue as Want to update terraform providers to a fresher version as provided in the configuration. but haven't outlined what simplified steps (what config? what terraform commands specifically?) to recreate the issue you saw (although hopefully your detailed workaround description might help a future reader).

In the Steps to reproduce section, I put the modification of version for the different providers used in my project.

The main command in question in this issue is terraform init -upgrade, that is used in my make recipe. In combination with terraform providers lock -platform=darwin_amd64 -platform=linux_amd64, so that the file ~/.terraform.lock.hcl at the root level of my project will contain the updated versions for Linux & Mac users. We use the 2 systems in the production team.

Could you try to describe the issue you're seeing (the bug in Terraform you are describing) with a simplified configuration so we might be able to reproduce the issue?

I have updated the providers on 5 different terraform project and I didn't face the problem described above in the 5 cases. I didn't manage to identify a common root cause, that can trigger the problem.

Here is a more concise description of the problem:

  1. Upgrade the provider version in your backend.tf file
  2. ASk Terraform0.14 to update the providers and make them available for Linux & Max users, with the commands: 
    terraform init -upgrade
terraform providers lock -platform=darwin_amd64 -platform=linux_amd64
  3. You might face the error 
    Error: Could not retrieve providers for locking

I will try to build a simple project in which I face this issue.... :crossed_fingers:

nantiferov commented 3 years ago

Hi @thomasboussekey Do you have TF_PLUGIN_CACHE_DIR variable set by any chance?

I found recently that if this var is set, terraform not fully updates lock file https://github.com/hashicorp/terraform/issues/27135#issuecomment-743998844

thomasboussekey commented 3 years ago

Hi @nantiferov Very good clue :clap: The environment variable TF_PLUGIN_CACHE_DIR is set in our context

I've searched on the Internet to see if we can enrich our configuration in order to cache providers for multiple sources. The most interesting link that I've found is here. But there doesn't seem to have a configure to do it

nantiferov commented 3 years ago

@thomasboussekey I really like this feature with lock files in general. But it definitely should be improved.

So far I was able to fix this issue for me by hiding TF_PLUGIN_CACHE_DIR during terraform init and terraform providers lock execution in wrapper script.

Wrapper logic:

apparentlymart commented 3 years ago

As others have said, the local cache directory does unfortunately prevent terraform init from fetching all of the checksums from upstream, because it skips the upstream registry request for any provider already in the cache. That is unfortunately a current limitation by design, due to some technical constraints as discussed in #27399.

If your intent is to run both terraform init and terraform providers lock then it'd be better to run them in the opposite order, so that the providers lock command can get an opportunity to fetch the checksums from upstream first, but I think you've done it the other way around in this case because you specifically want to lock the updated packages, not the currently-selected ones.

If that's true, then I wonder if we could address this use-case by adding a similar -upgrade command to terraform providers lock so that you could potentially do this process in a one-shot way:

terraform providers lock -upgrade -platform=darwin_amd64 -platform=linux_amd64

If your only goal was to get the lock file updated then it would then not be necessary to run terraform init at all. If you want to run other commands though, you could still run terraform init (with no arguments) after running the command above, in order to get what I think is the result you intended for what you showed in your makefile.

Inverting the two should work here because terraform providers lock never uses the cache directory; indeed, its primary purpose is to be able to gather checksums from upstream even in situations where there's a custom provider installation configuration in effect that would prevent terraform init alone from doing it.

If that would be an acceptable solution to your current use-case, I'd like to use this issue to represent improving terraform providers lock to have that additional option, which would then allow us to potentially address that without blocking on the broader design work over in #27264.

nantiferov commented 3 years ago

Thank you @apparentlymart for so detailed comment.

If your intent is to run both terraform init and terraform providers lock then it'd be better to run them in the opposite order

But in this case terraform providers lock command will fail with Error: Module not installed, cause there is nothing in .terraform folder, cause terraform init hasn't executed yet. I'm talking now about the first run situation, when there is a terraform code in folder, but nothing was run (e.g. terraform init, etc).

Inverting the two should work here because terraform providers lock never uses the cache directory

AFAIK it actually uses. In my case I was able to create a correct lock file with terraform providers lock only if TF_PLUGIN_CACHE_DIR was disabled for both terraform init and terraform providers lock. Otherwise, I'm getting this error:

$ echo $TF_PLUGIN_CACHE_DIR
/Users/user/.terraform.d/plugin-cache
$ unset TF_PLUGIN_CACHE_DIR
$ echo $TF_PLUGIN_CACHE_DIR

$ terraform providers lock -platform=darwin_amd64 -platform=linux_amd64
- Fetching hashicorp/github 4.1.0 for darwin_amd64...
- Obtained hashicorp/github checksums for darwin_amd64 (signed by HashiCorp)
- Fetching hashicorp/github 4.1.0 for linux_amd64...

Error: Could not retrieve providers for locking

Terraform failed to fetch the requested providers for linux_amd64 in order to
calculate their checksums: some providers could not be installed:
- registry.terraform.io/hashicorp/github: the current package for
registry.terraform.io/hashicorp/github 4.1.0 doesn't match any of the
checksums previously recorded in the dependency lock file.

I think that it would be nice to have an option in terraform init command to define what providers should be added to lock file.

My goals is that lock file is generated at the first run of terraform init with correct hashes and for both platforms.

xM8WVqaG commented 3 years ago

As others have said, the local cache directory does unfortunately prevent terraform init from fetching all of the checksums from upstream, because it skips the upstream registry request for any provider already in the cache. That is unfortunately a current limitation by design, due to some technical constraints as discussed in #27399.

Would it be possible to have a reliable way of disabling plugin-cache-dir configuration? For example, during terraform init -upgrade. Then upgrades can still be done in one command.

Using unset TF_PLUGIN_CACHE_DIR as @nantiferov suggests only works when the plugin-cache-dir is configured with an env var. The env var only takes precedence over the config file while it's not "". Setting it to /dev/null errors with mkdir /dev/null: not a directory and setting to a directory that doesn't exist (TF_PLUGIN_CACHE_DIR=false) just creates that directory (./false).

I was working around this with:

TF_PLUGIN_CACHE_DIR="" \
TF_CLI_CONFIG_FILE=terraform-upgrade.hcl \
tf -chdir=project init -upgrade

where ./terraform-upgrade.hcl looks like:

plugin_cache_dir=""

However, I've now had to temporarily disable provider locking. I'm maintaining dozens of terraform projects, all of them with lots of providers. Not using the cache means to even update a single provider across all projects blocks Terraform for a good 20 minutes while my slow home internet pointlessly redownloads each provider repeatedly for every project. Hopefully #27264 will be resolved soon.

thomasboussekey commented 3 years ago

Thank you @apparentlymart , @nantiferov and @xM8WVqaG for your contributions,

I worked on this issue and tried to implement @xM8WVqaG workaround, in my Makefile recipes and terraform wrapper. But, I didn't manage to make it work at the first time.

I will follow the issue #27264, expecting it will ease our provider's version management :crossed_fingers:

Have a nice week-end,

thomasboussekey commented 3 years ago

Eager to solve this issue, I managed to refresh the provider at the awaited version with a single GNU make recipe

refresh_providers:
    $(TERRAFORM) init -upgrade
    $(TERRAFORM) providers lock -platform=linux_amd64
    $(TERRAFORM) providers lock -platform=darwin_amd64 -platform=linux_amd64

Where ${TERRAFORM} calls a terraform wrapper that has the TF_PLUGIN_CACHE_DIR environment variable set to an existing folder. My CACHE dir only contains the linux versions (I execute my scripts from a linux server). It seems that darwin's versions are not cached.

Content of the TF_PLUGIN_CACHE_DIR folder

/home/tboussekey/.terraform.d/plugin-cache
├── linux_amd64
│   ├── CHANGELOG.md
│   ├── LICENSE
│   ├── LICENSE-3rdparty.csv
│   ├── README.md
│   ├── terraform-provider-aws_v2.70.0_x4
│   ├── terraform-provider-datadog_v2.20.0
│   ├── terraform-provider-google-beta_v3.52.0_x5
│   ├── terraform-provider-google_v3.52.0_x5
│   ├── terraform-provider-random_v2.2.1_x4
│   ├── terraform-provider-random_v2.3.1_x4
│   ├── terraform-provider-template_v2.2.0_x4
│   ├── terraform-provider-vault_v2.17.0_x4
│   └── terraform-provider-vault_v2.18.0_x4
└── registry.terraform.io
    ├── cyrilgdn
    │   └── postgresql
    │       ├── 1.11.0
    │       │   └── linux_amd64
    │       │       ├── CHANGELOG.md
    │       │       ├── LICENSE
    │       │       ├── README.md
    │       │       └── terraform-provider-postgresql_v1.11.0
    │       └── 1.9.0
    │           └── linux_amd64
    │               ├── CHANGELOG.md
    │               ├── LICENSE
    │               ├── README.md
    │               └── terraform-provider-postgresql_v1.9.0
    ├── datadog
    │   └── datadog
    │       ├── 2.18.1
    │       │   └── linux_amd64
    │       │       ├── CHANGELOG.md
    │       │       ├── LICENSE
    │       │       ├── LICENSE-3rdparty.csv
    │       │       ├── README.md
    │       │       └── terraform-provider-datadog_v2.18.1
    │       └── 2.20.0
    │           └── linux_amd64
    │               ├── CHANGELOG.md
    │               ├── LICENSE
    │               ├── LICENSE-3rdparty.csv
    │               ├── README.md
    │               └── terraform-provider-datadog_v2.20.0
    └── hashicorp
        ├── archive
        │   └── 2.0.0
        │       └── linux_amd64
        │           └── terraform-provider-archive_v2.0.0_x5
        ├── aws
        │   ├── 2.70.0
        │   │   └── linux_amd64
        │   │       └── terraform-provider-aws_v2.70.0_x4
        │   ├── 3.13.0
        │   │   └── linux_amd64
        │   │       └── terraform-provider-aws_v3.13.0_x5
        │   ├── 3.22.0
        │   │   └── linux_amd64
        │   │       └── terraform-provider-aws_v3.22.0_x5
        │   ├── 3.24.1
        │   │   └── linux_amd64
        │   │       └── terraform-provider-aws_v3.24.1_x5
        │   └── 3.25.0
        │       └── linux_amd64
        │           └── terraform-provider-aws_v3.25.0_x5
        ├── google
        │   ├── 3.51.0
        │   │   └── linux_amd64
        │   │       └── terraform-provider-google_v3.51.0_x5
        │   └── 3.53.0
        │       └── linux_amd64
        │           └── terraform-provider-google_v3.53.0_x5
        ├── random
        │   ├── 2.3.1
        │   │   └── linux_amd64
        │   │       └── terraform-provider-random_v2.3.1_x4
        │   ├── 3.0.0
        │   │   └── linux_amd64
        │   │       └── terraform-provider-random_v3.0.0_x5
        │   └── 3.0.1
        │       └── linux_amd64
        │           └── terraform-provider-random_v3.0.1_x5
        └── vault
            ├── 2.15.0
            │   └── linux_amd64
            │       └── terraform-provider-vault_v2.15.0_x4
            ├── 2.17.0
            │   └── linux_amd64
            │       └── terraform-provider-vault_v2.17.0_x4
            └── 2.18.0
                └── linux_amd64
                    └── terraform-provider-vault_v2.18.0_x4

48 directories, 45 files
solarmosaic-kflorence commented 2 years ago

Just ran into this on the latest terraform version (currently 1.0.10). In my case, this happens when developing on macOS locally and using the same code in automation in our Linux machines.

This makes updating provider dependencies painful, as now I have to run the following commands every time I want to update provider dependencies in a single folder (there are many folders):

Unfortunately if I don't use plugin cache, things can be really slow locally... so there really isn't a great workaround for me at the moment other than writing a script to do this for me.

solarmosaic-kflorence commented 2 years ago

Duplicated by https://github.com/hashicorp/terraform/issues/27810

sanyer commented 2 years ago

oh good, I thought I was alone in this

standage-thanh commented 1 year ago

Hi my friends! I have run into this issue. Could you please tell me the best workaround? I don't want to delete .terraform.lock.hcl file because it contains version info.