Add native support for updating .terraform.lock.hcl

This pull request addresses a performance issue for updating .terraform.lock.hcl at scale by introducing a new tfupdate lock command.

The dependency lock file (a.k.a. .terraform.lock.hcl) was introduced in Terraform v0.14. While it works fine for simple setups, updating hundreds of lock files scattered across multiple directories is still challenging.

At that time, I avoided implementing the feature for updating lock files in tfupdate because:

The lock file format is the implementation details of Terraform.
Its format appears to be extensible for modules in the future.
It's easy to imagine supporting multiple Terraform versions would be hard.

Therefore, I worked around it by using the official terraform providers mirror and terraform providers lock commands. https://github.com/minamijoyo/tfupdate/issues/32

I understood the root cause of this complexity is that the official Terraform Registry doesn't return h1 hash values, so I proposed changing the protocol of Terraform Registry in the upstream. https://github.com/hashicorp/terraform/issues/27264

However, the situation became even worse while we were waiting for progress in the upstream:

With the rise of arm chips, especially m1 mac, the number of platforms to calculate has increased.
As growing my infrastructure to be managed, the number of directories to calculate has increased.
Starting from Terraform v1.4, lock file management is substantially required to enable provider caching. This means that we can no longer have the final resort of throwing it into .gitignore. https://github.com/hashicorp/terraform/issues/32205

After over two years of waiting for progress in the upstream, it's time to bite the bullet and reimplement it myself, knowing it's the implementation details.

This PR adds a new tfupdate lock command which updates .terraform.lock.hcl without Terraform CLI.

$ tfupdate lock --help
Usage: tfupdate lock [options] <PATH>
Arguments
  PATH               A path of directory to update
Options:
      --platform     Specify a platform to update dependency lock files.
                     At least one or more --platform flags must be specified.
                     Use this option multiple times to include checksums for multiple target systems.
                     Target platform names consist of an operating system and a CPU architecture.
                     (e.g. linux_amd64, darwin_amd64, darwin_arm64)
  -r  --recursive    Check a directory recursively (default: false)
  -i  --ignore-path  A regular expression for path to ignore
                     If you want to ignore multiple directories, set the flag multiple times.

Note that unlike the terraform providers lock command, the --platform flag requires two hyphens (--).

The tfupdate lock command parses the required_providers block in your configuration, downloads provider packages, and calculates hash values under the hood. The most important point is that it caches calculated hash values in memory, giving us a huge performance advantage when updating multiple directories using the -r (--recursive) option.

$ tfupdate lock --platform=linux_amd64 --platform=darwin_amd64 --platform=darwin_arm64 -r ./

To skip terraform init, we assume that all dependencies are pinned to a specific version (e.g. 3.2.1) in the required_providers block of the root module. Note that version constraint expressions (e.g. > 3.0, = 3.2.1 )or indirect dependencies via modules are not supported and are ignored.

Closes https://github.com/minamijoyo/tfupdate/issues/68

minamijoyo / tfupdate

Add native support for updating .terraform.lock.hcl #90