"terraform providers add" subcommand for easier provider dependency additions

[apparentlymart]

Adding a new provider requirement to a Terraform configuration is a step typically done at least once for each module, since a Terraform module that doesn't use any providers is typically (but not always) of limited use.

Adding provider requirements in a robust way currently requires a number of manual steps:

1. Look up the provider in Terraform Registry to find out what its current version number is.
2. Add the required_providers block if this is the first provider dependency in the configuration, and then add an entry to that block mentioning the provider and setting a minimum or pessimistic version constraint for it.
3. Run terraform init to get the provider installed and generate the .terraform.lock.hcl entry for it.
4. (do all of the other things you intended to do with the provider...)

An idea from a long time ago that we didn't end up pursuing at the time was a CLI command to automate away some of these steps. For example:

$ terraform providers add hashicorp/aws

I imagine that this command would automatically perform actions functionally equivalent to steps 1 through 3 in my list above.

Of course, there are various details of this that we'd need to figure out, including but not limited to:

• If there isn't already a required_providers block somewhere in the configuration then we'll need to make a unilateral decision about where to place it.

  The upgrade tool for Terraform v0.13 established a convention of that living in a file called versions.tf, so I would propose we continue that same convention.

• If there is already a required_providers block then we should add a new entry to it. By default we should follow the typical convention that the local name for the provider matches the last part of its source address; for example, the local name for hashicorp/aws is conventionally aws. But in edge cases we might find there's already a provider using that local name, so we'll need a rule for making it unique.

  I propose trying to add 2 to the end, and then consecutive incrementing decimal integers until we find one that isn't already in use. If the command ends up doing this then it should emit an extra warning in the output about the local name not being the standard one, which means that any resources using this provider will need an explicit provider argument. Thankfully this situation does not arise very often.

• When we generate the required_providers entry we'll need to decide what kind of version constraint to use by default.

  We can't know if the current module is a root module or a shared module, so the safest option would be to use a >= constraint as we recommend for shared modules and let the user constrain it further to ~> or an exact constraint via manual editing if they want to. Either way we'd still be generating a .terraform.lock.hcl entry specifying exactly the version we found, so the version selection will be locked by default.

• When we generate the dependency lock file we'll need to generate checksums. As usual with today's registry protocol we'll probably only be able to generate zh: checksums for all platforms and have to calculate a h1: checksum locally for the current platform based on the package we downloaded. Future improvements to the provider registry protocol to allow provider developers to sign h1: checksums too will benefit terraform providers add in the same way as they would benefit terraform init.

---

One major motivation for making this an imperative command is to make it easier to include in provider setup instructions and in other tutorials. Today provider developers and tutorial authors often need to spend a lot of words explaining how to do something like the steps 1 through 3 above, and it would be nice to replace that with just a single command to run.

There is precedent for commands like this in other language ecosystems:

• Rust has cargo add
• Node's NPM has npm install --save
• Go has go get, which is similar but not quite the same

These examples show that there might be some other options we could consider adding to terraform providers add either initially or in later updates:

• A way to influence what kind of version constraint Terraform generates in the required_providers block.
• A way to add "dev dependencies" rather than normal dependencies, once that becomes a meaningful distinction in Terraform. (I expect it to become so once something like terraform test is finalized, so that one can somehow declare dependencies used only for testing.)
• A way to specify additional platforms to record checksums for, essentially subsuming some of the role of terraform providers lock for proactively generating h1: checksums targeting additional platforms.

[zetHannes]

It would be lovely if this was added. One question I'd have to the draft: If the default version should be specified with the >= constraint, how would the "correct" constrained version be determined?

[apparentlymart]

In the manual step 3, if Terraform finds a requirement for a provider that isn't currently present in the lock file then it selects the latest available version that matches the version constraints in the configuration.

In this more automated flow then, I expect that terraform providers add would find the latest available version for the provider and:

• Set the version constraint to >= that version.
• Set the exact selected version to that version, which will be recorded in the dependency lock file.

The first rule above is under the assumption that most authors won't immediately test their module against earlier versions of the provider anyway, and so it would be over-promising to choose any earlier version. Of course, this is only a bootstrapping helper so there's nothing to stop a module author from manually loosening the constraint and then testing with earlier versions of the provider, if they choose to.

One thing I notice in revisiting this proposal is that we'd be implicitly accepting the idea that even shared modules would end up with dependency lock files, but they'd be used only for direct development work such as running terraform test because dependency lock files are a whole-configuration concept that belongs in the root module. With the addition of terraform test this new possibility of shared modules having lock files seems inevitable to me anyway, but we should take some care to document this carefully to minimize confusion about how the dependency lock files of different modules will interact with one another. (The answer is: they actually won't interact at all, because only one dependency lock file is ever in scope at a time.)