Add support for `--terragrunt-source-map` as a more consistent, predictable way to override source URLs

[yorinasub17]

Terragrunt's --terragrunt-source parameter, which is designed to make it easy to override source URLs in terragrunt.hcl, has a couple issues:

1. If you run terragrunt apply --terragrunt-source /foo/bar in folder yyy, as of now, the only thing it does is override the source URL in yyy/terragrunt.hcl to the exact value /foo/bar. However, if you run terragrunt apply-all --terragrunt-source /foo/bar in folder yyy, it overrides the source value of each child module within yyy/sub/path to something like /foo/bar/sub/path. This inconsistency can be confusing and only gets worse when working with read_terragrunt_config, where we have to parse another terragrunt.hcl and decide whether to override its source URL or not.
2. --terragrunt-source assumes all your modules are coming from the same root repo. You won't be able to do an apply-all if you're pulling from multiple repos (e.g., your own infra modules, Terraform Registry, Gruntwork repos, etc).

Perhaps a better solution to this is to support some sort of mapping:

terragrunt apply --terragrunt-source-map source:dest

The --terragrunt-source-map param would replace any source URL that starts with source to instead start with dest.

For example:

terragrunt apply --terragrunt-source-map github.com/org/modules.git:/local/path/to/modules

The above would replace source = "github.com/org/modules.git//xxx" with /local/path/to/modules//xxx regardless of whether you were running apply, or apply-all, or using a dependency. This would give us a consistent behavior across all commands and support mapping multiple repos by specifying the param multiple times:

terragrunt apply \
  --terragrunt-source-map github.com/org/modules.git:/local/path/to/modules \
  --terragrunt-source-map github.com/org/another-repo.git:/local/path/to/another-repo

We could also consider allowing users to specify this mapping in a file:

terragrunt apply --terragrunt-source-map mapping.hcl

Where mapping.hcl contains:

mappings = {
  "github.com/org/modules.git"      = "/local/path/to/modules"
  "github.com/org/another-repo.git" = "/local/path/to/another-repo"  
}

See https://github.com/gruntwork-io/terragrunt/pull/1134#discussion_r407167726 for context for this issue.

[yorinasub17]

Core feature was implemented in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.29.3, but still missing the following:

• [x] Environment variable configuration (starting v0.29.5)
• [ ] file based configuration

[yorinasub17]

Env var config PR: https://github.com/gruntwork-io/terragrunt/pull/1676

[queglay]

I am wondering, can this handle the immutable ref suffix if used? https://github.com/gruntwork-io/terragrunt/issues/1627#issuecomment-847082445

in the example linked, there may be modules with these types of links:

source = "github.com/firehawkvfx/firehawk-main.git//modules/vault?ref=v0.0.20"
source = "github.com/firehawkvfx/firehawk-main.git//modules/vault-kv-init?ref=v0.0.20"
source = "github.com/firehawkvfx/firehawk-main.git//modules/vault-policies?ref=v0.0.20"

We would want to override all these to point at another path to test before the next release.

[yorinasub17]

Yes this should work with --terragrunt-source-map github.com/firehawkvfx/firehawk-main.git=/path/to/local/modules. The source map feature ignores and drops query parameters when doing the mapping.

[queglay]

Thank you for clarifying.

[ppozeti]

First of all this is a great feature! I've got a lot of nested modules and it is a pain to test them.

Sadly I got no luck configuring that in Windows. I tried to run like --terragrunt-source-map git::https://azuredevopsurl/_git/=C:\IaC and it fails with "Did not find any Terraform files (*.tf)".

I compared debug logs with/without the --terragrunt-source-map argument and it seems that the "WorkingDir" variable gets a non hashed value when using --terragrunt-source-map with C:\ in the path.

Have anyone being able to use this with Windows?

[davidwneary]

The source map feature ignores and drops query parameters when doing the mapping.

Are there any plans to allow this behaviour to be configurable? e.g. --preserve-query-parameters or equivalent?

My use case is that I define the paths to my modules using something like

git::ssh://git@gitlab.com/company/project.git?ref=1.0

This means a developer can check the project out and run a plan without any faffing with credentials or personal access tokens.

However, my CI runner doesn't have an SSH key, so would ideally use a path like

git::https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/company/project.git?ref=1.0

I tried using TERRAFORM_SOURCE_MAP for this but unfortunately, because the query parameters are ignored, it will always point to the master branch rather than the ref specified.

Cheers!

[angeloskaltsikis]

The source map feature ignores and drops query parameters when doing the mapping.

Are there any plans to allow this behaviour to be configurable? e.g. --preserve-query-parameters or equivalent?

My use case is that I define the paths to my modules using something like

git::ssh://git@gitlab.com/company/project.git?ref=1.0

This means a developer can check the project out and run a plan without any faffing with credentials or personal access tokens.

However, my CI runner doesn't have an SSH key, so would ideally use a path like

git::https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/company/project.git?ref=1.0

I tried using TERRAFORM_SOURCE_MAP for this but unfortunately, because the query parameters are ignored, it will always point to the master branch rather than the ref specified.

Cheers!

Can you try to add to your CI .gitconfig the following contents like Atlantis does https://github.com/runatlantis/helm-charts/blob/main/charts/atlantis/values.yaml#L53 ? I think this will solve your issue.

[davidwneary]

@angeloskaltsikis amazing! Thanks so much for the quick response, this did just the trick. I added the following as a pre-script:

git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf ssh://git@gitlab.com

And now no need to use TERRAFORM_SOURCE_MAP at all in the CI. Thanks a lot!

[angeloskaltsikis]

@angeloskaltsikis amazing! Thanks so much for the quick response, this did just the trick. I added the following as a pre-script:

git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf ssh://git@gitlab.com

And now no need to use TERRAFORM_SOURCE_MAP at all in the CI. Thanks a lot!

That's great to hear. I would be glad if you can add me in LinkedIn to discuss further about your CI and how do you use it :-)