Module Versioning

[apparentlymart]

(This was split out of #1439, to capture one of the use-cases discussed there.)

---

Currently all of the information for requesting a module gets packed into the source string, which cannot be parameterized:

module "foo" {
  source = "github.com/hashicorp/example"
}

Some of the supported module sources have an idea of versioning native to them. For example, the Github, Bitbucket and Generic Git sources can select from tags, branches, and specific commits by packing a refspec on the end in a pseudo-querystring:

module "foo" {
  source = "github.com/hashicorp/example?ref=v0.1.0"
}

This is problematic in an environment where users want to gradually shift between versions on different environments. For example, it's reasonable to want to run a newer version of a module in a staging environment than in production, so that the module can be tested.

To address this we need to make module versioning a first-class idea and provide mechanisms to work with versions such that they can vary between workspaces and be temporarily overridden for development.

For initial syntax we can potentially borrow from the design of versioned providers, added in 0.10:

module "foo" {
  source  = "github.com/hashicorp/example"
  version = "~> 0.1"
}

This allows specifying a "default" version, but it doesn't address the need for the version to vary between workspaces or be overridden temporarily for development.

Therefore we'll need to devise a mechanism to meet these needs.

Ideally this mechanism would also generalize to provider versions, since they too can benefit from varying between workspaces and in development environments to allow testing of new provider versions before using them in production.

[apparentlymart]

Here's one idea for how we might solve for this:

Introduce the idea of "lock files", taking inspiration from systems such as rubygems, that specify exact versions being used, separately from the configuration which specifies ranges of acceptable versions for automatic selection.

terraform init would by default look for a lock file whose name is systematically derived from the current workspace name. For example, terraform.d/default.tflock. If such a file is present, it is consulted for the versions of each referenced module. If it is not present, terraform init uses the constraints in configuration to select suitable versions and generates this lock file with the selected versions. (This is conceptually tricky, because terraform init isn't a workspace-sensitive command; we might need to tweak the workflow here to make this make sense.)

terraform init would also take a command line option to override the lock file location, allowing this mechanism to be used independently of workspace selection.

One challenge here is that we need to consider transitive module dependencies: module foo may depend on module baz, so the lock file must be able to express the version selected for foo.baz as well as just foo. This seems solvable with some careful design of the lock file format.

Committing the lock file to version control would be recommended, with its contents either being managed directly (for custom workflows) or just kept up to date using terraform init -upgrade.

The same lock file format could retain locks for both modules and plugins, thus addressing both needs with one mechanism. The precise dependencies are then saved in the configuration repository along with all of the other configuration, making it easier to reproduce the exact setup used to apply a given configuration.

(There is already a "lock file" for provider plugins in 0.10, but it serves a different purpose: it keeps track of exactly what binaries were present during terraform init, so that Terraform can detect changes and prompt to re-run terraform init. The lock file discussed in this issue would track versions using identifiers meaningful to the repository that the plugin is coming from, such as an official version number, whereas the existing lock file tracks the concrete SHA256 hash of the binary that was installed.)

---

Some further prototyping is definitely warranted here to look for any complexities in this workflow that I've not considered here, and to see how the UX of it feels in practice.

[rorychatterton]

I'm helping drive forward a POC around TF Enterprise 2 with Westpac in Australia. This is a feature we would be extremely interested in a future release.

Whichever way this is implemented, an additional feature around Terraform Registry for tagging may need to be considered, as this will become the primary way organisations expose their modules. (i.e. tag releases with nightly/pilot/prod and allow early adopters to integrate with new releases of a module).

@nedshawa - FYI

[apparentlymart]

Hi @rorychatt!

When I originally opened this the Terraform Registry wasn't really in the picture, but indeed it is planned to make this feature integrate with the versioning mechanism in Terraform Registry. The exact details of this are still being figured out but there should be some more info on this coming soon.

[jbardin]

Module versions are now available in master.

[bam0382]

What release is it a part of @jbardin ?

[jbardin]

@bam0382, this will be part of the 0.11 release.

[rorychatterton]

I noticed this release still doesn't allow you to pass version as a variable. Is there anything coming here as a future release?

[apparentlymart]

There are no plans to allow dynamically-set versions. The compatibility between modules is a property of the calling module code itself, so it doesn't make sense for it to be dynamically settable, and it's just not technically possible to do so due to the fact that module installation is a separate step in terraform init, which does not accept variables.

[roshan]

If I wish to hold the following invariant true, what is the preferred way to do things in Terraform?

1. Add two instances of a module (say two of this RDS module)

2. Ensure that both instances are using the same version of the module

Normally, what I'd expect is that I'd store the version in some variable and reference it in my two module directives. Then, when I want to upgrade to a newer version of the module, I change one place instead of finding and changing each instance. Is this something you don't intend to ever support or is this something that should be done another way?

[kainz]

Is this something that could be handled by a terraform override?

[wollerman]

@jbardin is this only supported for registry? Or is it possible to use the version flag as suggested in the proposal above, i.e.:

module "foo" {
  source  = "github.com/hashicorp/example"
  version = "~> 0.1"
}

Looking through the documentation I only see reference to versioning as mentioned in the original proposal under the terraform registry.

Thanks!

[jbardin]

@wollerman,

Yes, versioning is integrated with the module registry. We don't currently have any plans to extend the versioning system to other sources.

However, when using a source like github for example, you do still have the option to specify a specific commit or tag to ensure you are using a known version of the module.

[jgiles]

@jbardin it would be hugely helpful for us to be able to use semver constraints on github sources in order to automatically pick up patch fixes while avoiding breaking changes.

Should I open a separate issue with that feature request? We'd be happy to help adding the feature if the "no plans" statement is bandwidth-driven.

[jbardin]

Hi @jgiles,

Fell free to open a new issue for that feature. As for any timeline, we can review the idea after the next major release, once all the major changes have settled.

Thanks!

[lorengordon]

Personally I'd prefer interpolation for the entire source parameter. I'd rather like to pull all my source definitions to the top of a configuration, in a locals definition, so I don't have to go hunting through every file to find/update the string. I expect it would make modules much more maintainable overall.

[ghost]

A current workaround for that @lorengordon is using a Terrafile to download the modules locally http://bensnape.com/2016/01/14/terraform-design-patterns-the-terrafile/ - however I agree that having a similar alternative natively with Terraform would be a great addition

[lorengordon]

@cam-fulcrum Nice... I had read that once-upon-a-time but never got around to implementing it... I use terragrunt quite a bit... I can see how using a Terrafile in combination with the new terragrunt hooks (to actually fetch the modules) might meet my need... Having layers upon layers of modules (which I do rather a lot) is probably not practical, though, without being able to interpolate the source field.

[sgreben]

re this (@jgiles):

it would be hugely helpful for us to be able to use semver constraints on github sources in order to automatically pick up patch fixes while avoiding breaking changes.

I built some non-invasive tooling to help keep an overview / track updates for terraform modules: terraform-module-versions. It only reads/queries source and module repositories and does not require vendoring.

In particular, it also parses the version field for git(-hub) based modules as a semver range, to provide "is there an update" output. Example (markdown output):

$ terraform-module-versions -updates -pretty examples/main.tf

UPDATE?	NAME	CONSTRAINT	VERSION	LATEST MATCHING	LATEST
Y	example_git_ssh	~> 0.10	0.10.0	0.11.2	1.11.5
?	consul	> 0.1.0		0.7.3	0.7.3
(Y)	consul_github_ssh	0.1.0	0.1.0		0.7.3
	consul_github_https	0.7.3		0.7.3	0.7.3

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.