Custom installation strategy for for modules, with the possibility of local mirrors

When invoking init, plan and apply a parameter like --module moduleName=path/to/module should allow to avoid the lookup in the registry for sources. Similarly with a zip file: --module moduleName=path/to/module.zip

This would considerably speed up local testing and iterations since it would decouple the hardcoding of external modules.

Current Terraform Version

1.0.4

Use-cases

Decouple local module testing by non hardcoding other modules locations
Delegating to external build tools, such as Gradle or similar, the scaffholding of the project
In plugin Gradle Terraform Plugin it would allow source sets dependencies and module dependencies between external project, while also publishing modules as a Maven dependency.

Attempted Solutions

It's not really possible at the moment without hardcoding paths or urls.

Proposal

With a local directory path

Invoking terraform plan --module Guimove/bastion/aws=path/to/local/sources should not use the registry when executing:

module "bastion" {
    "source" = "Guimove/bastion/aws"
    ...
}

But instead it should look for it in given path path/to/local/sources.

With a local zip file

nvoking terraform plan --module Guimove/bastion/aws=path/to/local/sources.zip should not use the registry when executing:

module "bastion" {
    "source" = "Guimove/bastion/aws"
    ...
}

But instead it should look for it in given path path/to/local/sources.zip, extract it somewhere and use those sources.

Thanks for sharing this use-case, @lamba92!

I don't think we could design this exactly as you described because it would mean terraform plan gaining module installation responsibilities, whereas module installation today is intentionally limited only to terraform init.

However, when we were designing the provider installation settings we did also imagine having some similar mechanisms for customizing module installation in the future, perhaps including the possibility of something analogous to what Terraform calls a "filesystem mirror" or "network mirror" for a module that's normally hosted elsewhere.

We didn't pursue that functionality at the time because module installation is, in spite of the similar user-facing workflow, quite a different process from provider installation in Terraform today. For example:

A configuration as a whole either depends on a provider or not and selects a single version of that provider, whereas module dependencies are entirely separate for each call. We'd need to think about whether these overrides happen based on the source address (as you showed in your writeup here) or based on the call address, like module.bastion. For registry-based source addresses, there's also the question of whether an override is for the module source address directly or for a particular version of that module.
Providers always originate in provider registries, although we allow users to create local mirrors of providers from registries. Modules can be installed from various different source types in addition to module registries, and so we need to think about whether this would be supported and how it would look for non-module-registry source addresses.

I expect we won't be able to pursue this further in the near future because we recently had a few departures from our team and we're applying our more-limited resources to other problems at the moment. However, we do intend to revisit module installation in the medium term, and at that time we'll definitely consider use-cases like this one. Thanks again!

Indeed I would agree that plan should not be responsible for retrieving modules, but it should be at least aware of the locations of the local ones either implicitly through some default location or explicitly through command line.

I would also not recommend to have mixed responsibilities in the code itself. The code for declaring module and providers versions should be separated from the actual source code. That would also allow to ditch the lock file, since you would be able to easily specify the version of the module/provider just like Maven or Gradle.

As such, with the given limitations of the current terraform module system, I've moved as much setup logic as possible to the Gradle plugin. At the moment I have to rely on rebuilding the execution context every time, but I managed to support the publication of a zip file with resources and typesafe resource access.

The plugin is also able to manage multiple source sets in the same project and to inherit modules built with the plugin and published on a Maven repository or another Gradle project. Have a look at the example here.

I would also be interested in a solution like this, perhaps for a more generic use case.

When publishing Terraform modules, a best practice is to include examples/foo/main.tf demonstrating how the module should be used.

When providing E2E testing of the module, you can either terraform init from the root, or (even better) from each of the examples/. A GH action that runs terraform init, fmt, and validate within each example would help keep the module clean.

Within each examples/foo/main.tf you may have a block like:

module "foo" {
  source = "publisher/modulename/provider//modules/submodule" # where "//modules/submodule" indicates a submodule
}

The problem with this is that there is no way to hint that the version should be the local version rather than the latest from the publisher path.

The alternative of using ../ source paths makes the examples testable, and usable from a git clone, but not usable as a copy/paste from the example path (which also serves as instructions).

Perhaps a terraform init argument like -plugin-dir for modules could help.

$ cd examples/foo
$ terraform init -module-dir=publisher/modulename/provider=../../

Thego.mod replace patterns could also be used for inspiration.

@apparentlymart I do see that development workflow story is lacking in terraform : local testing, integration testing, unit-testing with provider mock-ups(biggest challenge).

About this particular ask, we've a near-good-ish solution and any feedback is here more than welcome

TL;DR: The way we are solving this problem is by over-writing the output of tf-init stage We wrote a wrapper on top of terraform that basically inject overrides the module

IMHO I'd like to see some improvement in Case-2 described below. i.e if somehow we can prevent downloading of modules and generate a pre-download pre-module.json and we can override it with our custom source, and then resume tf-init that would be great

In development workflow there are two cases that we tried to answer

Testing of local module (newly written, not used by anyone) : --local-overrride capirca="/my/symlink/dir/capirca"
```
module "capirca_local" {
  # for local testing only
    source = "../local/capirca"
    ...
}
```
In this case, our wrapper-cli basically symlink all the files in development dir of module into terraform-stagedir/local. This happen before even tf-init is called. If your dev workflow do not have a concent of stage dir then you can skip this case.
Testing of module for any update already in use and published. (integration testing). --module-override fb.com/core/capirca=my/custom/local/dir/capirca
```
module "capirca_remote" {
  source = "fb.com/core/capirca"
  version = "0.1.2"

   ...
}
```
Sometimes we have to run plan against particular service(cloud-account already provisioned with tf code) to see if changes made in module is backward compatible or not.

So, what we do is bascially abuse tf-init output. It does need to download modules from original sources eg. "fb.com/core/capirca" in this case .

tf-init then generates a module.json file with a mapping of module sources. We simply edit this file and replace all modules which has Source==fb.com/core/capirca.

total 4
drwxr-xr-x 1 sonukr users   66 Aug 18 04:32 .
drwxr-xr-x 1 sonukr users  376 Aug 18 04:32 ..
drwxr-xr-x 1 sonukr users   58 Aug 18 04:31 modules
drwxr-xr-x 1 sonukr users   54 Aug 18 04:32 providers
-rw-r--r-- 1 sonukr users 1618 Aug 18 04:31 terraform.tfstate

module.json file generated by tf.init

{
  "Modules": [
    {
      "Key": "",
      "Source": "",
      "Dir": "."
    },
    {
      "Key": "capirca",
      "Source": "fb.com/core/capirca",
      "Version": "1.0.21",
      "Dir": ".terraform/modules/capirca"
    },
    {
      "Key": "sk_capirca",
      "Source": "fb.com/core/skdummy_fb_capirca/internal",
      "Version": "1.0.414",
      "Dir": ".terraform/modules/sk_capirca"
    }
  ]
}

$cloud terraform plan <service> plan/apply --module-override  fb.com/core/capirca="s3://a/b/c/my-super-awesome-.rc.zip"
  -> stage all tf files
  -> run tf-init
  ->  DO THE MAGIC HERE
  -> run tf plan or tf-apply

Psuedo code:DO THE MAGIC HERE

At this phase what we do is once tf-init done downloading all the modules and providers , it generates a modules.json file.
Look for module-override option and downloads the module to a tmp dir, say /tmp/my-awesome-module
Edit module.json and replace Dirwith /tmp/my-awesome-module wherever Source==fb.com/core/capirca

Hi all,

I recently turned a newer issue into effectively a duplicate of this one because I forgot that this one was already open. Sorry about that!

Since I've already been discussing potential technical designs over in that other issue I'm making the unusual decision to consolidate the older issue into the new one rather than the other way around. If you're still interested in this proposal, please subscribe to and vote :+1: on https://github.com/hashicorp/terraform/issues/31134 instead! And if you have any feedback on the design question I posed in my most recent comment over there (whether it's acceptable to limit this only to registry-style source addresses) then I'd love to hear your perspective.

Thanks!

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform