Terragrunt can't find outputs in dependency

gercograndia commented 2 months ago

Describe the bug

A while ago, we upgraded to terragrunt version v0.58.2 in an azure environment, and were already noticing things were quite a bit slower than before.

But last week we ran into a confusing issue, where a particular module coulnd't resolve the outputs of a dependency, while it was definitely there.

ERRO[0910]  221:     dbxuc_schema_system = dependency.databricks_config.outputs.uc_audit.schema.id
ERRO[0910]
ERRO[0910] This object does not have an attribute named "outputs".

It must be said: this module had (by nature) quite a few dependencies, but all of them resolved fine except this one.

Inspecting the output of the said module showed that the outputs were definitely there. However, the terragrunt output command takes about 3 minutes, while just running plain terraform output from the .terragrunt-cache directory completes almost instantly (less than a second).

The debug log mentioned that the dependency is skipped because it is disabled.

time=2024-08-02T14:45:39+02:00 level=debug msg=Skipping outputs reading for disabled dependency databricks_config

The dependency declaration itself didn't have a enabled attribute set, but the terragrunt.hcl of the dependency does have a skip statement.

Anyways, while testing earlier versions of terragrunt we noticed the following:

v0.55.1 is the last version that is both performant as well as properly resolving the dependencies.
both v0.55.2 and v0.55.3 has the described issue, but are still 'quick'.

in v0.55.5 another strange error popped up, but the plan is what we would expect:

ERRO[0008]  110:   groups = dependency.ad_groups.outputs.azuread_groups
ERRO[0008]
ERRO[0008] There is no variable named "dependency".

in v0.55.6 that aforementioned error is gone, but the main issue (can't find the outputs) popped up. Also, it became very slow from this release onwards.

We tried also some newer versions but that wasn't helpful either:

In latest version (v0.66.1) we got the following error:

ERRO[0167]   on xxxxx/terragrunt.hcl line 67:
ERRO[0167]   67:         [for g in try(v.member_of, []) : try(dependency.ad_groups.outputs.azuread_groups[g].id, null)]
ERRO[0167]
ERRO[0167] with
ERRO[0167] v.member_of set to null
ERRO[0167] .
ERRO[0167] A null value cannot be used as the collection in a 'for' expression.
ERRO[0167] Error: Iteration over null value

v0.60.x no new issues, but the main 'outputs' issue was still there.

We didn't put more effort into the newer versions yet, we might do that. We're also more than willing to test out a few things if that's going to be helpful.

Our CICD plan route decreased from 36 minutes (using v0.58.3) to 10 minutes (using v0.55.1)

Steps To Reproduce

Just changing the terragrunt version and run terragrunt plan.

Expected behavior

It should resolve the outputs (and not be that slow :-)

Versions

Terragrunt version: v0.58.3
OpenTofu/Terraform version: terraform v1.5.7
Environment details (Ubuntu 20.04, Windows 10, etc.): macos

Additional context

n/a

gercograndia commented 2 months ago

Maybe in addition:

I reduced the terragrunt.hcl to remove all other dependencies so that only the 'faulty' dependency remained.

However, the issue still remained, so I guess it has not so much to do with the sheer number of dependency declarations.

kjonca commented 2 months ago

Seems we are also hit by this issue (terragrunt 0.66.1). Interesting thing is: when I temporarily change to "skip=false" in dependency, then run "terragrunt plan" (which I believe fetches the output of dependency), then, after restoring to "skip=true" - output is visible and is used.

gercograndia commented 2 months ago

Seems we are also hit by this issue (terragrunt 0.66.1). Interesting thing is: when I temporarily change to "skip=false" in dependency, then run "terragrunt plan" (which I believe fetches the output of dependency), then, after restoring to "skip=true" - output is visible and is used.

Hmm interesting. Let me do some tests here as well.

However, I woud argue that if you have a skip = true in a dependent module, then it would make sense to not fetch the outputs right? I must say that I am not very keen on this skip option, it gives quite a few issues.

kjonca commented 2 months ago

"dependent module, then it would make sense to not fetch the outputs right? " It's complicated. :) We use this to make some configuration tests. And skips is used to tell "we do not want to test this big and complicated module" now, but we want to check modules which depends on it.

levkohimins commented 2 months ago

@gercograndia, @kjonca, Guys, can you provide the simplest possible configuration to reproduce the issue?

gercograndia commented 2 months ago

Hi @levkohimins,

I'll see if we can reproduce it (tomorrow) in a as simple as possible setup but in our case it will never be easy.

I think the situation is as follows:

module_1 has a dependency on module_2 and wants to use the outputs.
module_2 has a skip attribute, and uses the output of a dependency to module_3 to decide whether or not to skip.

I think the situation of @kjonca is different: trying to get hold of the outputs when skip = true.

levkohimins commented 2 months ago

Thanks @gercograndia, If you can't provide a simple configuration, that's not a problem, as long as it reproduces the issue.

gercograndia commented 2 months ago

Here you go @levkohimins

Assuming both module-2 and module-3 are applied, run a terragrunt plan in module-1.

Tested with terragrunt v0.58.3, which shows the issue (just as the latest, v0.66.1). v0.55.1 is doing it fine.

terragrunt-issue-3321.zip

What this config does NOT show though, is the huge performance degradation introduced in v0.55.6.

levkohimins commented 2 months ago

Thank you very much @gercograndia. Let's solve the "no outputs" issue first, and then I'll try to look at the performance.

gercograndia commented 2 months ago

Great, thanks a lot in advance.

gruntwork-io / terragrunt