[maybe] Terraform Dependency Management Broken

[ekristen]

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

No response

If you're self-hosting Renovate, select which platform you are using.

None

If you're self-hosting Renovate, tell us what version of the platform you run.

GitHub

Was this something which used to work for you, and then stopped?

It used to work, and then stopped

Describe the bug

Looks like terraform provider and modules source dependency management is broken.

Ultimately it looks like something with some caching mechanism is causing mismatches in dependencies and branch update failures.

The behavior appears to be that the renovate dashboard is updated with WARN: Error updating branch: update failure and reviewing the logs it seems to be mismatching which dependency it should update and ends with update failure

In the logs provided, it's trying to update the azuread provider, but it seems to resolve to the local provider which is also defined in the versions.tf

Looks like this broke 3 days ago. Jan 9, 22:18:00Z

• 971327268
• 971327522 (this one is really broken)

Relevant debug logs

Logs

``` DEBUG: Starting search at index 191(packageFile="versions.tf", branch="renovate/azuread-2.x") { "depName": "azuread" } DEBUG: Found match at index 191(packageFile="versions.tf", branch="renovate/azuread-2.x") { "depName": "azuread" } DEBUG: depName mismatch(packageFile="versions.tf", branch="renovate/azuread-2.x") { "manager": "terraform", "currentDepName": "azuread", "newDepName": "local" } ```

Have you created a minimal reproduction repository?

No reproduction repository

[ekristen]

I hate to ping you @rarkins but this looks to be pretty broken. I can reproduce on at least 4-5 repositories, each new one I try ends up with the same result.

[github-actions[bot]]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[rarkins]

Needs a reproduction or some type of before/after logs with a noticeable difference.

[ekristen]

@rarkins nothing has changed on my end. It appears that renovate is mismatching dependencies leading to branch update failures.

Here's another log number 971114781.

DEBUG: manager.getUpdatedPackageFiles() reuseExistingBranch=true(branch="renovate/github.com-cloudposse-terraform-aws-key-pair-0.x")
DEBUG: Rebasing branch after deps list has changed(packageFile="main.tf", branch="renovate/github.com-cloudposse-terraform-aws-key-pair-0.x")
{
  "depName": "github.com/cloudposse/terraform-aws-key-pair"
}
DEBUG: manager.getUpdatedPackageFiles() reuseExistingBranch=false(branch="renovate/github.com-cloudposse-terraform-aws-key-pair-0.x")
DEBUG: Starting search at index 7050(packageFile="main.tf", branch="renovate/github.com-cloudposse-terraform-aws-key-pair-0.x")
{
  "depName": "github.com/cloudposse/terraform-aws-key-pair"
}
DEBUG: Found match at index 7050(packageFile="main.tf", branch="renovate/github.com-cloudposse-terraform-aws-key-pair-0.x")
{
  "depName": "github.com/cloudposse/terraform-aws-key-pair"
}
DEBUG: depName mismatch(packageFile="main.tf", branch="renovate/github.com-cloudposse-terraform-aws-key-pair-0.x")
{
  "manager": "terraform",
  "currentDepName": "github.com/cloudposse/terraform-aws-key-pair",
  "newDepName": "github.com/my-org/tf-cloudinit"
}

[ekristen]

This started Jan 9th, 2023 at 22:18Z. I'll try and throw a repo together real quick and share.

[ekristen]

Also it looks like I can't access any logs older than 3 days ago on the dashboard, so I'm out of luck getting you any before logs.

[ekristen]

@rarkins a brand new repository seems to work ok, so could the cache be bad? It's affecting every single one of my private repositories I've tested so far that renovate been configured on for months. I'm up to 10 so far.

[ekristen]

https://github.com/ekristen/renovate-bug-terraform <-- worked just fine, but I have 10 + logs now showing it not working on other repositories.

[rarkins]

Are these open branches/PRs? If so, is the result the same when you tick rebase/retry?

[ekristen]

@rarkins sorry for the ping, from my viewpoint every repository I had that I checked was affected, glad it appears to not be a global thing at the moment.

It's still broken for multiple repositories of mine, here's another log ID 971383821 and this one was just a couple minutes ago 971383845

In the log 971383821 (sanitized a bit for the comment)

This would be a PR/branch.

DEBUG: syncBranchState()(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: getBranchPr(renovate/github.com-my-org-tf-azure-linux-0.x)(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: findPr(renovate/github.com-my-org-tf-azure-linux-0.x, undefined, open)(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: findPr(renovate/github.com-my-org-tf-azure-linux-0.x, undefined, closed)(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: branchExists=false(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: dependencyDashboardCheck=undefined(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: recreateClosed is false(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: findPr(renovate/github.com-my-org-tf-azure-linux-0.x, chore(deps): update terraform github.com/my-org/tf-azure-linux to v0.5.3, !open)(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: prAlreadyExisted=false(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: Checking schedule(at any time, null)(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: No schedule defined(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: Branch needs creating(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: Using reuseExistingBranch: false(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: Setting current branch to master(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: latest commit(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
{
  "branchName": "master",
  "latestCommitDate": "2022-10-16T11:02:21-06:00"
}
DEBUG: manager.getUpdatedPackageFiles() reuseExistingBranch=false(branch="renovate/github.com-my-org-tf-azure-linux-0.x")
DEBUG: Starting search at index 5256(packageFile="o.tf", branch="renovate/github.com-my-org-tf-azure-linux-0.x")
{
  "depName": "github.com/my-org/tf-azure-linux"
}
DEBUG: Found match at index 5256(packageFile="o.tf", branch="renovate/github.com-my-org-tf-azure-linux-0.x")
{
  "depName": "github.com/my-org/tf-azure-linux"
}
DEBUG: depName mismatch(packageFile="o.tf", branch="renovate/github.com-my-org-tf-azure-linux-0.x")
{
  "manager": "terraform",
  "currentDepName": "github.com/my-org/tf-azure-linux",
  "newDepName": "github.com/cloudposse/terraform-null-label"
}
WARN: Error updating branch: update failure(branch="renovate/github.com-my-org-tf-azure-linux-0.x")

[ekristen]

This is what the dashboard looks like. For log 971383821

[ekristen]

That looks about right from what I can tell. The after is what I'm seeing all over the place right now, mismatching dependency names leading to updating branch failures.

I just tried rebasing as you requested. Log is 971383971. I ticked the box for chore(deps): update terraform github.com/sans-sroc/tf-azure-linux to v0.5.3 and it's still ticked after renovate ran, it ended with dep mismatch.

[rarkins]

{"level":30,"renovateVersion":"34.94.0","msg":"Repository started","time":"2023-01-09T18:06:56.891Z"}
...
{"level":20,"branch":"renovate/aws-3.x","msg":"syncBranchState()","time":"2023-01-09T18:07:06.582Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"branch.isUpToDate(): using cached result \"true\"","time":"2023-01-09T18:07:06.583Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"getBranchPr(renovate/aws-3.x)","time":"2023-01-09T18:07:06.583Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"findPr(renovate/aws-3.x, undefined, open)","time":"2023-01-09T18:07:06.583Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Found PR #94","time":"2023-01-09T18:07:06.583Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"branchExists=true","time":"2023-01-09T18:07:06.583Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"dependencyDashboardCheck=undefined","time":"2023-01-09T18:07:06.583Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"PR rebase requested=false","time":"2023-01-09T18:07:06.583Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Checking if PR has been edited","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"branch.isModified(): using cached result \"false\"","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Found existing branch PR","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Checking schedule(at any time, null)","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"No schedule defined","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Branch already exists","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"getBranchPr(renovate/aws-3.x)","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"findPr(renovate/aws-3.x, undefined, open)","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Found PR #94","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Skipping behind base branch check due to rebaseWhen=auto","time":"2023-01-09T18:07:06.584Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"isBranchConflicted(master, renovate/aws-3.x)","time":"2023-01-09T18:07:06.585Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"branch.isConflicted(): using cached result \"false\"","time":"2023-01-09T18:07:06.585Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Branch does not need rebasing","time":"2023-01-09T18:07:06.585Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Using reuseExistingBranch: true","time":"2023-01-09T18:07:06.585Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Checking if we can automerge branch","time":"2023-01-09T18:07:06.585Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"mergeStatus=no automerge","time":"2023-01-09T18:07:06.585Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Ensuring PR","time":"2023-01-09T18:07:06.585Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"There are 0 errors and 0 warnings","time":"2023-01-09T18:07:06.585Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"getBranchPr(renovate/aws-3.x)","time":"2023-01-09T18:07:06.585Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"findPr(renovate/aws-3.x, undefined, open)","time":"2023-01-09T18:07:06.586Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Found PR #94","time":"2023-01-09T18:07:06.586Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Found existing PR","time":"2023-01-09T18:07:06.586Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Fetching changelog: https://github.com/hashicorp/terraform-provider-aws (3.74.3 -> 3.76.1)","time":"2023-01-09T18:07:06.586Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Processing existing PR","time":"2023-01-09T18:07:06.886Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Pull Request #94 does not need updating","time":"2023-01-09T18:07:06.906Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"PR is not configured for automerge","time":"2023-01-09T18:07:06.907Z"}

{"level":30,"renovateVersion":"34.97.1","msg":"Repository started","time":"2023-01-10T21:11:11.330Z"}
...
{"level":20,"branch":"renovate/aws-3.x","msg":"syncBranchState()","time":"2023-01-10T21:11:31.564Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"branch.isUpToDate(): needs recalculation","time":"2023-01-10T21:11:31.564Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"getBranchPr(renovate/aws-3.x)","time":"2023-01-10T21:11:31.564Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"findPr(renovate/aws-3.x, undefined, open)","time":"2023-01-10T21:11:31.564Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Found PR #94","time":"2023-01-10T21:11:31.565Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"branchExists=true","time":"2023-01-10T21:11:31.565Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"dependencyDashboardCheck=undefined","time":"2023-01-10T21:11:31.565Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"PR rebase requested=false","time":"2023-01-10T21:11:31.565Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Checking if PR has been edited","time":"2023-01-10T21:11:31.565Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"branch.isModified(): using cached result \"false\"","time":"2023-01-10T21:11:31.565Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Found existing branch PR","time":"2023-01-10T21:11:31.565Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Checking schedule(at any time, null)","time":"2023-01-10T21:11:31.565Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"No schedule defined","time":"2023-01-10T21:11:31.565Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Branch already exists","time":"2023-01-10T21:11:31.566Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"getBranchPr(renovate/aws-3.x)","time":"2023-01-10T21:11:31.566Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"findPr(renovate/aws-3.x, undefined, open)","time":"2023-01-10T21:11:31.566Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Found PR #94","time":"2023-01-10T21:11:31.566Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Skipping behind base branch check due to rebaseWhen=auto","time":"2023-01-10T21:11:31.566Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"isBranchConflicted(master, renovate/aws-3.x)","time":"2023-01-10T21:11:31.566Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"branch.isConflicted(): using cached result \"false\"","time":"2023-01-10T21:11:31.566Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Branch does not need rebasing","time":"2023-01-10T21:11:31.566Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Using reuseExistingBranch: true","time":"2023-01-10T21:11:31.578Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"Setting current branch to master","time":"2023-01-10T21:11:31.579Z"}
{"level":20,"branch":"renovate/aws-3.x","branchName":"master","latestCommitDate":"2022-11-17T10:27:21-07:00","msg":"latest commit","time":"2023-01-10T21:11:31.749Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"manager.getUpdatedPackageFiles() reuseExistingBranch=true","time":"2023-01-10T21:11:31.816Z"}
{"level":20,"branch":"renovate/aws-3.x","packageFile":"versions.tf","depName":"aws","msg":"Rebasing branch after deps list has changed","time":"2023-01-10T21:11:32.521Z"}
{"level":20,"branch":"renovate/aws-3.x","msg":"manager.getUpdatedPackageFiles() reuseExistingBranch=false","time":"2023-01-10T21:11:32.522Z"}
{"level":20,"branch":"renovate/aws-3.x","packageFile":"versions.tf","depName":"aws","msg":"Starting search at index 96","time":"2023-01-10T21:11:32.584Z"}
{"level":20,"branch":"renovate/aws-3.x","packageFile":"versions.tf","depName":"aws","msg":"Found match at index 96","time":"2023-01-10T21:11:32.585Z"}
{"level":20,"branch":"renovate/aws-3.x","manager":"terraform","packageFile":"versions.tf","currentDepName":"aws","newDepName":"local","msg":"depName mismatch","time":"2023-01-10T21:11:32.619Z"}
{"level":40,"branch":"renovate/aws-3.x","msg":"Error updating branch: update failure","time":"2023-01-10T21:11:32.620Z"}

[ekristen]

It seems that the packageFile index is somehow locating the incorrect dependency match?

[rarkins]

Release notes: https://octoclairvoyant.vercel.app/comparator?repo=renovatebot%2Frenovate&from=34.94.0&to=34.97.1

@secustor I guess this is related to the HCL parser change

[rarkins]

@ekristen if you haven't had a commit to the main branch since then, it's possible that a new commit would clear things out

[ekristen]

Interesting. It does seem like a cache problem, so cache prior to HCL parser change vs after especially since it doesn't seem to affect a branch new repository (ie https://github.com/ekristen/renovate-bug-terraform/issues/4)

Let me give the commit bump a try.

[rarkins]

FYI I just deleted the cache for tpl-tf-aws-range and enqueued a job

[ekristen]

@rarkins sounds good, I was just about to bump the commit on that one. I'll hold off or try another one.

[rarkins]

That seems to have fixed it for that repo, but I don't know which cache caused the problem. @ekristen if you can find a reason to commit to main branch for a different repo - which should invalidate the extract cache - then we will be closer to narrowing it down

[ekristen]

I'm bumping tpl-tf-azure-range now.

[ekristen]

Looks like 971381307 was the log for that bump and everything is happy now?

[rarkins]

The changes to test files in #19269 should have invalidated the extract cache, but it seems that maybe didn't happen. I'm trying to test this manually from CLI now

[rarkins]

@RahulGautamSingh this appears to be an extract cache validation problem.

I run against github.com/renovate-reproductions/terraform1 with 34.94.0 with cache enabled. Then run again against 34.95.0 and it's still valid

[ekristen]

That was my assumption when I first opened the issue is that there was a bad cache thing going on based on the logs, that's why I didn't think to do a reproducing repository. I'll make sure to do that next time regardless.

My concern was that it was affecting more than just me originally. I suspect it probably still is, but maybe not as bad as it originally appeared to me.

Thanks again for being awesome @rarkins (and the whole team), what you've built here is amazing. I appreciate the hard work and the wonderful tool you've given us.

[rarkins]

Thanks for reporting and your help troubleshooting. I think the root cause is that we are not invalidating the extract cache when a manager's source changes, so we need to address that to fix the problem

[rarkins]

@RahulGautamSingh looks like the manager fingerprints aren't being used for extract - only for update. I forget if that was a deliberate decision or if I even made it for some reason, but it seems we need to revisit it. Maybe I thought that invalidating the branch cache would be enough but clearly it's not.

[rarkins]

I think I see the problem. If we use the hashMap as part of the extract fingerprint too simply, then we will invalidate extract caches for any manager change. Instead, we should incorporate it only for managers which have any matching files.

[RahulGautamSingh]

I think including manager fingerprint in extract fingerprint will solve this issue. It seems I forgot to add it there. My bad.

I think I see the problem. If we use the hashMap as part of the extract fingerprint too simply, then we will invalidate extract caches for any manager change. Instead, we should incorporate it only for managers which have any matching files.

I don't think its possible as we won't have the fileList until we clone the repo.

[rarkins]

We could also cache the list of managers which had matching files in the past run, and then the combined fingerprint of those. Then we can reuse that list without cloning.

[woehrl01]

Hi @rarkins ,

I also experience that issue in the last days, with the regex manager, will your proposed fix address this, too?

Is there anything I can share here with you?

[rarkins]

if it's the same issue, then any commit to the base branch fixes it. If it doesn't, then it's not the same issue

[woehrl01]

@rarkins thanks, then it's a different issue with the same depName, newDepName mismatch error. I'll create a new issue with more information.

[renovate-release]

:tada: This issue has been resolved in version 34.119.0 :tada:

The release is available on:

• GitHub release
• 34.119.0

Your semantic-release bot :package::rocket: