Cannot downgrade or pin the terraform provider version

Gerrit-K commented 2 years ago

This might rather be a question or starting point for discussion instead of an issue, but I hope it's still the right place to address it.

Issue description

When using the prebuilt providers, such as the azurerm provider, cdktf generates the required_providers section like this:

"azurerm": {
  "version": "~> 2.0",
  "source": "azurerm"
},

This essentially causes terraform to always fetch the latest 2.0 provider, no matter what the prebuilt provider package was built against. Currently there seems to be no way to manually change that value to downgrade or pin the provider.

Background

This is especially problematic for environments where you need reproducible builds. I.e., in our application we deliberately use a yarn.lock (as probably many other teams do as well; same with package-lock.json) to pin working configurations in order for CI/CD pipelines to "replay" them consistently. Pinning there currently still does not pin the actual provider that is used.

To give a real example with impact: our pipelines started to fail last friday due to a bug in the 2.86 release, although we didn't change anything in the configuration. As a temporary workaround, we now had to revert back to the generated provider instead of the prebuilt one to pin it to version 2.85, which, unfortunately, also required quite some code changes in the import statements.

Fix / Suggestion

It would be great if either:

One could somehow override the required_providers field to a custom version, or
The prebuilt provider always pins to the exact terraform provider it was built against, e.g. 0.2.10 to 2.78.0. This would allow the users to control the provider version via package.json and the corresponding lock files. But one downside is that it would also tightly couple the release cycle of the prebuilt provider to the upstream terraform provider (if that's not already the case?).

Looking forward to hear feedback and opinions on that.

drdamour commented 2 years ago

@Gerrit-K have you looked into https://www.terraform.io/docs/language/dependency-lock.html ?

Gerrit-K commented 2 years ago

@drdamour good point! :+1: I didn't ... that would definitely be the way to go if we used terraform alone. However, since we use cdktf, all terraform files are generated dynamically to ./cdktf.out/stacks/<stackname>/, which we exclude from VCS, but that would exactly be the place to put the dependency lock file into, wouldn't it? Did I overlook something? Is there another way to use this lock file using cdktf?

jsteinich commented 2 years ago

I'm in favor of the build provider containing exactly the version that it was built against. See https://github.com/hashicorp/terraform-cdk/issues/318

github-actions[bot] commented 2 years ago

This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

Gerrit-K commented 2 years ago

Not stale (at least for now)

@jsteinich since my second suggestion is essentially the same as what you've already proposed in the main repo, I would be fine with closing this as a (semi-)duplicate of that one. WDYT?

ansgarm commented 2 years ago

@Gerrit-K: As Jon reacted with a thumbs up to your message, I'm going to close this 👍

We'll use https://github.com/hashicorp/terraform-cdk/issues/318 to track this from now on 👀

Gerrit-K commented 2 years ago

@ansgarm ah yes, sorry for the lack of response. Thanks for closing it :) I already subscribed to the other one ✅

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for at least 7 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue so we can investigate further.

cdktf / cdktf-provider-azurerm