search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.45k stars 9.51k forks source link

Ability to disable lockfiles #31533

Open wyardley opened 2 years ago

wyardley commented 2 years ago

Current Terraform Version

1.2.6

Use-cases

We use Renovate to pin versions of basically everything, and we .gitignore lockfiles. However, the Terraform feature to create / use a lockfile is still not configurable.

This means that when planning locally, one has to remember to use terraform init -upgrade when versions have changed -- even if you are already pinning versions to explicit known versions that you want. It also creates unnecessary lockfiles in cases where the lockfiles are gitignored and new environments are created.

Please reconsider the previously dismissed feature request to allow users to choose not to use a lockfile.

Attempted Solutions

Proposal

Terraform should offer an option to not use the lockfile at all when initializing / planning.

References

crw commented 2 years ago

Thanks for filing this request!

apparentlymart commented 2 years ago

Hi @wyardley!

Can you say a little more about how you are using Renovate?

I ask because my understanding of Renovate is that its purpose is to automatically propose upgrading to newer versions of a dependency, and for Terraform in particular I would've expected it to achieve that by proposing a change to the lock file which selects a newer version. It sounds like you're achieving that in a different way, in which case I'd like to hear more about it just so we can understand the problem underlying your request.

I don't expect we will be able to do exactly what you are requesting -- Terraform relies on its own lock file to track which provider versions you've selected, so Terraform cannot operate at all without it -- but I'd like to think about other ways we could help Renovate to integrate with Terraform.

wyardley commented 2 years ago

I ask because my understanding of Renovate is that its purpose is to automatically propose upgrading to newer versions of a dependency, and for Terraform in particular I would've expected it to achieve that by proposing a change to the lock file which selects a newer version. It sounds like you're achieving that in a different way, in which case I'd like to hear more about it just so we can understand the problem underlying your request.

So, there may be some rudimentary lockfile support at this point, though IIRC, it can only do it at a single top level lockfile maybe, and there have been some reported issues with certain providers...

https://github.com/renovatebot/renovate/issues/10550e

That said, I'm not sure it's fair for a tool like that to be able to support any specific arbitrary version of each tool's lockfile format -- to do so, it would either need to use Terraform, or manage exact compatibility with its lockfile format. For me, this is predictable enough - we update all instances of pinning terraform itself one version at a time, and the same for the various providers (GCP, AWS, Azure), across all states / substates / modules at a time, typically planning them, and validating in CI, to make sure things still are working as expected. I'm not sure if there could be a corner case where a module may pull in a transitive dependency that might change unexpectedly without using a lockfile, but I am totally Ok with that.

For all intents and purposes, maintaining the exact versions we want, and no lockfile, provides essentially the same behavior as using a tool to update those same versions and a lockfile at the same time, but with (IMO) much less hassle, and less risk of the tool messing up the lockfile, having issues with certain providers, etc.

It is typical (and for languages like Javascript or Ruby, which have a lot of dependencies and nested dependencies, we do follow this behavior) for Renovate to update the Lockfile as well. But with any of these languages, its default behavior is to pin all the dependencies it manages. With terraform, that basically means that, if you don't leave in any fuzzy version constraints, Renovate is already basically ensuring fairly predictable versions if you run terraform init -- with or without a lockfile present.

Terraform relies on its own lock file to track which provider versions you've selected, so Terraform cannot operate at all without it

I will obviously defer to you on what Terraform "needs", but I guess I would say that it's more accurate to say that Terraform needs the dependencies specified in its configuration to know which provider versions you've selected. It can use the lockfile to ensure some predictability between runs once it's created a lockfile, if used (and you all suggest, but do not require, checking in the lockfile to vcs), but since Terraform can initialize and create a lockfile when one is not present, and knows what versions to install, I would say that, even if it's not possible now, it should theoretically be possible to create a situation where Terraform doesn't create, or read, a lockfile, and, of course, this is how Terraform worked for many years.

In fact, as we're normally planning from a fresh environment with no lockfile or cache, we're already essentially behaving as if there were no lockfile, except in the occasional case we happen to plan something locally.

I fully expect that the answer is that I shouldn't want what I'm asking for, and that's fine.

ohookins commented 2 years ago

We are in a similar situation - we build a master image with Terraform and install all of our providers into it at fixed versions (previously using terraform bundle, now using terraform providers mirror). This is driven by a versions file that Dependabot can work with and automate our provider upgrades - thus the versions are already locked implicitly.

It would be great if we could tell Terraform to just use the versions that are present in the directory it will already be searching for providers, not to download anything that is missing, and critically - not to complain about this very valid mode of operation.

apparentlymart commented 2 years ago

I guess my ideal model of how Renovate should behave with Terraform is to only update the lock file, and to propose selecting the highest-precedence provider version (per semver rules) which meets the constraints specified in the configuration. The constraints in the specification are intended to represent what versions a module is compatible with, whereas the lock file represents the single version of each provider a provider is currently using within the bounds of those constraints.

I will admit that I'm not well versed in the details of how Renovate does its work, but it could achieve that by behaving as if it were performing the following steps:

Maintainers can then use the guidance in Understanding Lock File Changes to review the proposed change with consideration to their organization's typical policy for adopting new software.

From what you've described it sounds like Renovate doesn't directly run the package manager whose lock files / manifests it is updating and instead attempts to manipulate those files directly itself. Is that true? If so, then indeed that does seem like a troublesome thing to maintain over time, but for any manifest or lock file that is supposed to be included in version control the package manager must presumably remain backward-compatible with its own historical formats so that it can handle upgrading, and for Terraform in particular I would not expect us to be changing the dependency lock file parser in a way that would not accept files generated by older versions.

If the Renovate team is interested we'd love to learn about what's in the way of Renovate doing something which would achieve the same result as what I described above (even if it isn't implemented exactly how I described it).

Also, I guess I should address the observation that Terraform used to work without a dependency lock file. It is true that before the "dependency lock file" was instead a hidden manifest file under the .terraform directory and therefore typically folks had it in .gitignore as a side-effect of ignoring that directory, but Terraform did still have something with a similar purpose as the dependency lock file, and this manifest (wherever it's stored) is an important part of making sure that terraform apply and other similar commands are guaranteed to use whatever versions terraform init reported selecting, or produce an error if that isn't possible.

If you put .terraform.lock.hcl under .gitignore (or similar) then you can recreate the same effect as older Terraform versions which generated an equivalent file in a location that was also typically under .gitignore. As the original comment observes, Terraform will still "remember" in a particular work tree which versions the most recent terraform init selected; that's by design and terraform init -upgrade (or, alternatively, just deleting the lock file altogether) is the existing way to opt out of that behavior.

wyardley commented 2 years ago

I guess my ideal model of how Renovate should behave with Terraform is to only update the lock file, and to propose selecting the highest-precedence provider version (per semver rules) which meets the constraints specified in the configuration.

Renovate (by default) partially works by pinning all the dependencies (it can use semver, but its default is to pin everything, and with terraform, I don't think it handles fuzzy version specs perfectly anyway). It does this even with systems like npm, gem, etc. that use a Lockfile.

So therefore, you're getting a PR to update =1.1.1 to, say, =1.1.2, which is why the lockfile isn't adding much that's new. This is easier for a human to read and review vs. a generated lockfile, and creates the same amount of noise in terms of PRs to update versions in either case, so I'm not sure why the mode of keeping a fuzzy version spec in the main version spec and a strict one in the lockfile is better.

I will admit that I'm not well versed in the details of how Renovate does its work, but it could achieve that by behaving as if it were performing the following steps:

Clone the repository containing the Terraform configurations of interest. In each configuration working directory, run terraform init -upgrade to select the newest available versions that meet the constraints specified in the configuration.

It absolutely could do that, but I guess my point is that if one is already pinning all, or almost all, dependencies, the work needed to do this (the tool has to have the terraform binary baked into its container, potentially multiple versions of it) seems like a lot of work vs. just trusting that, since you asked for 1.1.1, you got 1.1.1. Presumably any concerns about checksum not matching or other security issues are already handled outside of the lockfile itself. So, I guess the question I have is, if you're already pinning all of your versions to exact equality, what additional benefit is the lockfile providing?

Additionally, this is extra work to resolve conflicts in the lockfile if (as they frequently are) they're introduced; Renovate already handles this seamlessly, but just seems kind of unnecessary here.

but Terraform did still have something with a similar purpose as the dependency lock file, and this manifest (wherever it's stored) is an important part of making sure that terraform apply and other similar commands are guaranteed to use whatever versions terraform init reported selecting

Perhaps, but in the Good Olde Days (TM), terraform would also give you the version you asked for if you updated it, and IIRC, not checking in that lockfile was the recommended (vs. non-recommended now) behavior.

If you put .terraform.lock.hcl under .gitignore (or similar) then you can recreate the same effect as older Terraform versions which generated an equivalent file in a location that was also typically under .gitignore

Yes, as mentioned, we already do that. So Terraform already is behaving this way (starting with no lockfile) not only when Renovate runs, but also in CI runs where validation happens, and so far, the sky hasn't fallen. And yet, it would be faster and simpler if terraform didn't need to create the lockfile (which will never be consumed) in this case.

The ask here is to allow terraform a way to ignore the lockfile (or always behave as if terraform init -upgrade were chosen), as well as to suppress creating a lockfile at all in situations where we know the lockfile won't persist.

And yes, I imagine I could add -upgrade to TF_CLI_ARGS_init for the local issue, and maybe I'll just do that.

Of course we can argue about the "right" way, or the "Hashicorp approved" way to do things; I guess what's frustrating is that it doesn't seem like it would introduce too many issues to create an option to prevent terraform from writing a lockfile in cases where it won't be read / used.

apparentlymart commented 2 years ago

I suppose where I'm getting caught up with this discussion is that we added the lock file specifically because people found it annoying that the version constraints in the configuration were simultaneously representing both "what dependencies are compatible with this module?" and "what exact version are we using right now?" such that people would get themselves into "dependency hell" situations where each of their modules was specifying a single exact version and they all had to always exactly agree in order for anything to work.

The introduction of a user-visible lock file, rather than one that Terraform maintains only as an implementation detail in a hidden directory, separated those two concerns so that modules could focus only on describing what they are compatible with, and the lock file can take over the responsibility of tracking what's currently selected.

From what you are saying it sounds like in your particular case you have no interest in tracking what modules are compatible with. I can imagine that being true if you are using exclusively first-party modules (so you don't need to deal with other people's version constraints) and you're motivated to stay on latest and just fix things when they break, rather than (as was true for the folks who advocated for lock files) treating upgrades as a project to be planned for and done explicitly at a convenient time.

I can certainly understand that position, but I'd prefer to find an answer where you can get what you want without the need for a special "use the modules themselves to track selected versions" mode.

Here's an question that might shed light on what I'm missing here: if Renovate hypothetically already had perfect support for updating the lock file as I described earlier, and if you just removed all of the version constraints from your modules entirely and used the lock file exclusively to track what versions you have selected (so Renovate would then always select the latest version of every provider to propose), what would you be missing?

So far having exact version constraints in your modules feels functionally equivalent to having exact versions tracked in the lock file, but I assume there must be some detail I'm not seeing! :grinning:

Regarding terraform init -upgrade, that -upgrade option has been required to get Terraform to consider newer versions of already-selected modules in the current working directory for as long as I can remember. As far as I know, all that we changed with the dependency lock file was putting that metadata in a prominent location where it could also be tracked in version control so that it could be shared between everyone working on a configuration, rather than each working directory having its own local selected versions.

As you've noticed yourself, TF_CLI_ARGS_init=-upgrade is one possible way to make that option "sticky" so that terraform init would just always upgrade things, regardless of what might've been previously suggested. But it sounds like you're dissatisfied with that answer... do you have an example of something that you'd consider more palatable to get that same effect?

wyardley commented 2 years ago

Here's an question that might shed light on what I'm missing here: if Renovate hypothetically already had perfect support for updating the lock file as I described earlier, and if you just removed all of the version constraints from your modules entirely and used the lock file exclusively to track what versions you have selected (so Renovate would then always select the latest version of every provider to propose), what would you be missing?

Absolutely -- in a perfect world, I'd have no issue with this situation (though in my example, I'd prefer to have the version specs still pinned and updated in the .tf files at the same time).

The problem is with "perfect" - I'm assuming Hashicorp doesn't have time to maintain / provide support for perfect dependency updates in Renovate (as well as comparable tools like Dependabot), nor to provide libraries in language other than golang to provide alternative interfaces besides the terraform binary for doing so, and it's also difficult to expect them to maintain perfect support for the various supported versions of terraform. Renovate does bake some binaries into its tools, but (even with things that are much more widely used than Terraform), the results are not always pefect.

If someone is going to devote the time to maintain supporting updating the lockfiles in a way that will continue to work (even if the lockfile version format changes in the future), I will definitely be enthusiastic about embracing the approach you are suggesting.

From what you are saying it sounds like in your particular case you have no interest in tracking what modules are compatible with. I can imagine that being true if you are using exclusively first-party modules (so you don't need to deal with other people's version constraints) and you're motivated to stay on latest and just fix things when they break, rather than (as was true for the folks who advocated for lock files) treating upgrades as a project to be planned for and done explicitly at a convenient time.

That's exactly right. I absolutely understand the utility of the lockfile for most people's use cases. However, when mostly or exclusively using first-party modules, in a single project along with the rest of the terraform code, it's as easy, or maybe easier, to update / check everything in lockstep in one fell swoop.

Regarding terraform init -upgrade, that -upgrade option has been required to get Terraform to consider newer versions of already-selected modules in the current working directory for as long as I can remember.

I'll have to play around with some old versions. It's been a while.

wyardley commented 2 years ago

So far having exact version constraints in your modules feels functionally equivalent to having exact versions tracked in the lock file, but I assume there must be some detail I'm not seeing!

Maybe with the format in the tf lockfile it's not as bad as with some, but reading diffs of generated lockfiles tends to be a lot harder / messier, in my experience, as is resolving conflicts. For many types of generated files, GH doesn't even show the diff by default.

image
wyardley commented 2 years ago

ps - I will try this out (checked in lockfiles w/ Renovate) with a small personal project and see how the current state is in terms of how it's working.

wyardley commented 2 years ago

So, the good news is that Renovate does deal with multiple lockfiles Ok, and it basically works, though the hash format seems different from what I generated locally [side note: zh: is referred to as legacy hash format here, but seems to be what I get by default with the latest terraform version on OS X.]

but, as suspected, the diff for the lockfile is both not displayed by default (ever) in the GH UI, and is somewhat harder to read (esp. in this case, where the hashes are turning over):

image image

Hopefully this helps explain why I don't like only the lockfile changing, especially, as well as some of my concerns about implementation varying depending on the logic of the program updating the lockfile and / or version of terraform used. This is not so hard when you're just updating one or two, but we have a single repo with our modules, and terraform configs for multiple projects / states, so a PR with version updates could easily include, let's say 30 or 60 of these updates.

wyardley commented 2 years ago

ps - If I init based on the lockfile Renovate proposes locally, it also changes the lockfile back as shown (maybe related to #27811; also related: https://github.com/renovatebot/renovate/discussions/11821):

diff --git a/gcp/.terraform.lock.hcl b/gcp/.terraform.lock.hcl
index 6f9ab2a..fe125f4 100644
--- a/gcp/.terraform.lock.hcl
+++ b/gcp/.terraform.lock.hcl
@@ -16,5 +16,17 @@ provider "registry.terraform.io/hashicorp/google" {
     "h1:mCRW9OtukK3KDG1lRyFIun2LuJue56QgnCru8kmeyPA=",
     "h1:n4dCdx9EhvoQO/SiLKZGkGKWxpCHodL5cjFtYDQhOQY=",
     "h1:vyXvM/V6cv7Ux2JO1Zd6IinJYutBVTgBK+PXga4fkoE=",
+    "zh:03fa16d2811fc3ef523b8afad5ce4c1c72e686a425f48f9432f21b16c55c2872",
+    "zh:23ad507dad0b3478c46b050f9d4660251e20b2ae6f334f4904af9aeef7e6f5d2",
+    "zh:2a20c8c6bb1c8185c7c4b6e418cbdb36ab212c549ffa7ab4c027f929224798bc",
+    "zh:53b8cd6ecc73c6fc1570dfbf9a549af975741c25cf544d6fec13d96c7312bd93",
+    "zh:5856c11c35636c362e04ddaf3cce7ed94e0aea74f22b6d3fc2449f1c0ce4b159",
+    "zh:68cda07855d20984d4fce6959a760392b1ef4e09e4f8e74bf10765b020727f0c",
+    "zh:84b501635c135a692f378dec2e56beca088d25f5e6c98276dbab8de30d27ce5e",
+    "zh:972e990c88f1b4c9bfe501e3a829c02bd89a494b3b445900216906365397a672",
+    "zh:a47acfb0b07d83687fab00275381755817aacec6f9f949e8d52e0bcae45e870f",
+    "zh:dcb078b2be1d8527e7bf9283112ba1214bb76d2279d78d921bebd413fb0310e0",
+    "zh:e00a1b11f0782e389b02c8a4487b2f25c42647714adcb56770f87883440997e9",
+    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
   ]
 }
wyardley commented 2 years ago

ps: https://github.com/renovatebot/renovate/issues/13692 is also well worth reading, and has some explanation / discussion on why they implement the lockfile update in code vs. using the terraform binary. For one thing, it seems like as things stand now, Renovate can't trivially use the terraform binary directly for this, though I'm guessing -backend=false might work for the situation where it can't init [edit: commenter in #27161 seems to mention that this doesn't work for this use case?]?

apparentlymart commented 2 years ago

Thanks for sharing these examples, @wyardley!

One specific potential improvement I can see from what you've shared is in how terraform init deals with a situation where the lock file already contains sufficient h1: checksums but doesn't include any of the h1: checksums.

As you've noticed, zh: is a legacy checksum format that we use to deal with the fact that the current provider registry protocol predates Terraform support other installation methods. There are various messy details here, but the short version is that a zh: is a checksum of the zip file that the author published, rather than the contents of the zip file, and so a checksum of that form isn't useful for anything other than verifying a package directly installed from the origin registry.

The newer h1: checksum format is more general because it's calculated from the files inside the package rather than from the package itself, and so Terraform can calculate an equivalent checksum both from a zip file and from an already-extracted directory.

Terraform currently has some logic to opportunistically "upgrade" from the zh: format to the h1: format during installation, to ensure that after installation Terraform will be able to verify both the extracted packages in the local package cache (under .terraform) and, if configured, packages retrieved from alternative sources like local filesystem directories or network mirrors.

It seems like that "upgrade" behavior also currently works in the opposite direction, effectively "downgrading" h1: packages to their equivalent zh: packages when installing from a registry. But it isn't really necessary to do that, because h1: checksums are already sufficient to verify packages obtained from all sources, including the origin registry.

So with all of this said: I think we should consider making Terraform not generate additional zh: checksums if there is already a h1: checksum sufficient to verify whatever was downloaded. And in practice there must be such a checksum in order to get this far, because otherwise terraform init would've failed with a checksum error while verifying the downloaded package.

Of course that doesn't address all of what you observed here, but it does at least seem like it would address this checksum-related oddity discussed in renovatebot/renovate#13692, without requiring Renovate to also generate the legacy checksum format.

apparentlymart commented 2 years ago

Separately, it also seems like Renovate is running into hashicorp/terraform#27161 and with the new context here, I can see that it was opened directly in response to Renovate's challenges using terraform init as I described earlier.

In that other discussion I recommended to use terraform providers lock, and I guess Renovate is now probably using that since that could explain why it generates lock files that contain only h1: hashes... terraform providers lock exists specifically to help with generating that new hash type in spite of the fact that the provider registry protocol doesn't support it.

Since there's already an issue open for that part I'll let that one continue to represent the question of how to get the lock file updated in a way that doesn't require backend initialization, which would be another part of the story of making Terraform support the "ideal" upgrade-detection flow I described earlier.

wyardley commented 2 years ago

As you've noticed, zh: is a legacy checksum format that we use to deal with the fact that the current provider registry protocol predates Terraform support other installation methods.

Yes, it was a little odd that running terraform init -upgrade locally using the latest version of Terraform generates a "legacy" format when installing from an existing lockfile (and in fact, removes the "new" / correct) version. I think that's basically what the existing issue is about.

In that other discussion I recommended to use terraform providers lock, and I guess Renovate is now probably using that

Without looking at the code, based on the discussion in the ticket, I think they're actually currently implementing it directly in code vs. shelling out to Terraform, but could be wrong there.

wyardley commented 2 years ago

Ok, had to look up the syntax for the old style config, but:

% docker run --platform=linux/amd64 --entrypoint=/bin/sh -it hashicorp/terraform:0.11.0
/ # cd /tmp/
/tmp # cat >> test.tf << EOF
> terraform {
>   required_providers {
>     aws = "1.0.0"
>   }
> }
> EOF
/tmp # terraform init

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
/tmp # sed -i.bak "s/1\.0\.0/1.0.1/" test.tf
/tmp # cat test.tf
terraform {
  required_providers {
    aws = "1.0.1"
  }
}
/tmp # terraform init

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
/tmp # ls -altr
total 16
drwxr-xr-x    1 root     root          4096 Aug 17 04:53 ..
-rw-r--r--    1 root     root            59 Aug 17 04:54 test.tf.bak
-rw-r--r--    1 root     root            59 Aug 17 04:54 test.tf
drwxrwxrwt    1 root     root          4096 Aug 17 04:54 .
/tmp # TF_LOG=debug terraform init
2022/08/17 04:54:57 [INFO] Terraform version: 0.11.0  ec9d4f1d0f90e8ec5148f94b6d634eb542a4f0ce+CHANGES
2022/08/17 04:54:57 [INFO] Go runtime version: go1.9
2022/08/17 04:54:57 [INFO] CLI args: []string{"/bin/terraform", "init"}
2022/08/17 04:54:57 [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2022/08/17 04:54:57 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2022/08/17 04:54:57 [INFO] CLI command args: []string{"init"}
2022/08/17 04:54:58 [DEBUG] command: loading backend config file: /tmp
2022/08/17 04:54:58 [INFO] command: empty backend config, returning nil
2022/08/17 04:54:58 [DEBUG] command: no data state file found for backend config
2022/08/17 04:54:58 [DEBUG] New state was assigned lineage "fa0f2e07-8d4a-43ab-b6d3-1081db14e665"
2022/08/17 04:54:58 [INFO] command: backend initialized: <nil>
2022/08/17 04:54:58 [DEBUG] checking for provider in "."
2022/08/17 04:54:58 [DEBUG] checking for provider in "/bin"
2022/08/17 04:54:58 [DEBUG] checking for provisioner in "."
2022/08/17 04:54:58 [DEBUG] checking for provisioner in "/bin"
2022/08/17 04:54:58 [INFO] Failed to read plugin lock file .terraform/plugins/linux_amd64/lock.json: open .terraform/plugins/linux_amd64/lock.json: no such file or directory
2022/08/17 04:54:58 [INFO] command: backend <nil> is not enhanced, wrapping in local
2022/08/17 04:54:58 [DEBUG] checking for provider in "."
2022/08/17 04:54:58 [DEBUG] checking for provider in "/bin"

Terraform has been successfully initialized!

2022/08/17 04:54:58 [DEBUG] plugin: waiting for all plugin processes to complete...
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Maybe I'm missing something, but I don't see any files / directories being created that persist after terraform init runs (there is an error about a plugin lockfile not existing), and I don't see a complaint when I adjust the version and reinitialize. No difference if I use a newer provider version listed in the registry that should be compatible with this version (2.70.0).

wyardley commented 2 years ago

Just for fun, also tried with latest 0.12.31; best I can tell, it's still not required to run terraform init -upgrade when updating versions:

% docker run --platform=linux/amd64 --entrypoint=/bin/sh -it hashicorp/terraform:0.12.31
/ # cd /tmp/
/tmp # cat >> test.tf <<EOF
> terraform {
>   required_providers {
>     aws = {
>       version = "3.75.1"
>     }
>   }
> }
> EOF
/tmp # terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 3.75.1...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
/tmp # sed -i.bak "s/3\.75\.1/3.75.2/" test.tf
/tmp # terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 3.75.2...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
/tmp # cat .terraform/plugins/linux_amd64/lock.json 
{
  "aws": "bf97a5ffb2ab43e033c8eab132a445249b9af3737f1cc017dffeee68f9e1bf85"
}/tmp #
apparentlymart commented 2 years ago

Hi @wyardley,

Thanks for sharing those examples. I see what you are getting at now.

In your most recent example I see that you found the lock.json file I was alluding to earlier, which is capturing a checksum that's equivalent to the zh: for linux_amd64 (albeit serialized a bit differently).

However you are right that in these older versions before the lock file, because in those days the version constraints were overloaded to represent both "what are we compatible with" and "what exactly are we using", changing the version constraints in the module behaved like how editing the lock file would behave today, and so indeed terraform init has no separate record to refer to about what version was currently selected and so had to rely exclusively on the configuration to get that information.

However, we considered that a design error: Terraform should not just quietly upgrade something without announcing that it did so, because that may cause the configuration to behave in a subtly different way or -- in the worse case -- end up running a compromised version of the plugin that doesn't match the checksums that a team originally reviewed and approved.

The -upgrade option is the mechanism we retained to allow opting out of this "locking" behavior, but it was an important requirement to be as secure as possible by default and require opting out of Terraform's integrity-checking features. I don't expect that we will add any other features to allow disabling those behaviors, beyond what's already present.

However, what we can do (and I think we should do) is prioritize fixing the remaining friction that makes it hard to use the lock file for its intended purpose of capturing the currently-selected provider versions, so that there is no longer any reason to misuse version constraints as if they were a lock file. This discussing has identified a few different things to work on already, and I've copied those into an internal list of possible future projects for our team to consider. The goal of that project would be to make the lock file work as intended by addressing these and other rough edges, so that there will be fewer reasons to ignore the lock file.

secustor commented 2 years ago

Hi, I have come across this thread by chance on one of our issues at Renovate and wanted to clear up some things regarding our implementation.

Renovate has an option to update the lockfile only if possible (`rangeStrategy: 'update-lockfile')

Yes, we use custom code to create the hashes and update the lockfile. This has multiple reasons:

@apparentlymart I would be happy to discuss a way how to simplify this and improve the experience for users of Renovate and Terraform.

apparentlymart commented 2 years ago

Hi @secustor! Thanks for sharing that extra context.

For the backend credentials issue at least, I'd suggest trying terraform init -upgrade -backend=false, where -backend=false instructs Terraform to not initialize the backend, and should therefore not require any backend credentials. This -backend=false option is there primarily to allow for credential-less CI workflows which run sequences like terraform init -backend=false && terraform validate (since validate is also guaranteed to never require credentials), but I believe it should be possible to combine it with -upgrade to get the effect of updating the lock file.

With that said, I also don't think there's any great harm in updating the lock file directly if Renovate has access to information that Terraform normally doesn't. The lock file format is a compatibility constraint and so future versions of Terraform should be able to consume the current lock file format for a long time, even if we were to introduce new versions of it in later releases.

The lock file includes the aggregated version constraints from the configuration tree specifically to allow other consumers to decode just the lock file and not have to deeply analyze the whole module tree to collect dependencies. (Terraform itself doesn't actually need to use that information, because it already knows how to walk the module tree to produce the same information.) That would be primarily aimed at what Renovate calls the update-lockfile strategy, I suppose.

With the intention that the constraints in the configuration are describing what the modules are compatible with rather than what they are currently selecting, and with our existing recommendation that module authors constrain the upper bound only if they already know a specific later version of the provider broke compatibility, it seems less useful to automatically update a module to remove its upper version constraint bound, but of course there's no real harm in supporting that as an option for those who have a different workflow e.g. involving only internal modules that can reasonably be updated all together, without any need to coordinate between modules published by different authors.

We would still like to make a new revision of the provider registry protocol which supports both the legacy hashing scheme and the current hashing scheme, but changing a documented protocol is of course a more significant effort than just changing implementation details inside the one CLI executable and also interacts with provider publishing workflows and therefore means rolling out a change across every separate provider repository.

ohookins commented 1 year ago

I tried to do what the helpful warning suggested and created the lockfile:

│ To calculate additional checksums for another platform, run:
│   terraform providers lock -platform=linux_amd64
│ (where linux_amd64 is the platform to generate)

However, we already have these providers cached in our image and this causes Terraform to reach out again to the registry in order to download the providers, which is not ideal - we download and pin the versions of the providers intentionally to avoid having to reach out to the registry, and also to have a known good image to run Terraform from that isn't running un-vetted code.

Regardless, this then causes the terraform init command to fail:

Initializing provider plugins...
- Reusing previous version of hashicorp/tls from the dependency lock file
- Reusing previous version of hashicorp/http from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v4.34.0...
- Installed hashicorp/aws v4.34.0 (unauthenticated)
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/tls: the previously-selected version 4.0.3 is no longer available
╵

╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/http: the previously-selected version 3.1.0 is no longer available
╵

I don't particularly feel like debugging this immediately, but I feel it's put us in a difficult position of having to work around some new behaviours or suffer endless noisy warnings.

apparentlymart commented 1 year ago

Hi @ohookins,

If you intend to lock the checksums in your local mirror directory, instead of the ones in the origin registry for each provider, then you can use the -fs-mirror=... option to instruct Terraform to use the filesystem mirror.

This particular command defaults to using the origin registry because the common use for it is to lock the checksums signed by the provider author (which are only available from the origin registry) and then verify a local mirror against those checksums later to make sure that the mirror is correct. However, in your case it seems like you would prefer to treat the mirror itself as the root of trust, so you can use -fs-mirror=... to tell Terraform that.

ohookins commented 1 year ago

The works great, @apparentlymart , thanks a bunch!

ohookins commented 1 year ago

Seems like I spoke too soon, and only ran it on a workspace that didn't have any of our private local modules. There's a bit of a conflict when trying to do this:

terraform providers lock -platform=linux_amd64 -fs-mirror=/usr/share/terraform/providers/

│ Error: Module not installed
│
│   on ecr.tf line 9:
│    9: module "ecr_image_cleanup" {
│
│ This module is not yet installed. Run "terraform init" to install all modules required by this configuration.
╵

But of course running terraform init is the step that complains about a missing lock file, which is generated on the previous step which can't run without the modules having been initialized.

pauldraper commented 7 months ago

I want to disable the lockfile because I don't use Terraform for provider resolution.

I compile the provider, copy to the right location, and run Terraform.

All this futzing around with lockfiles and their problems is entirely pointless. (At least in my case -- I'm sure it important if you're using third party providers and such.)