[Terraform Projects] Bump to Terraform 1.9.x

dduportal commented 3 months ago

Service(s)

AWS, Azure, Datadog, DigitalOcean, Update center, Other

Summary

We are using the 1.6.x line of Terraform today: https://github.com/jenkins-infra/packer-images/blob/d7c995afe5e637226d24362a673260ba8d76a03b/provisioning/tools-versions.yml#L45

This issue is to track the work associated to upgrading to Terraform 1.9.x:

[x] Check changelog(s) (add link to 1.7, 1.8 and 1.9)
- Terraform 1.7: https://github.com/hashicorp/terraform/blob/v1.7/CHANGELOG.md
- No breaking changes
- Focus on terraform test (which do not use..yet?) and terraform graph
- New features of interest of us: removed and import blocks
- Bug fixes
- Terraform 1.8: https://github.com/hashicorp/terraform/blob/v1.8/CHANGELOG.md mentioned and upgrade guide (new!) at https://developer.hashicorp.com/terraform/language/v1.8.x/upgrade-guides
- We might see "empty" changes during the first apply: https://developer.hashicorp.com/terraform/language/v1.8.x/upgrade-guides#possible-spurious-changes-when-refreshing no worries!
- We'll have to keep a look on the azure-net json export as per https://developer.hashicorp.com/terraform/language/v1.8.x/upgrade-guides#minor-change-to-jsonencode-function-results (but should not be an issue)
- New features which could be interesting: around provider locks and new built in providers for encoding/decoding tfvars files
- Terraform 1.9: https://github.com/hashicorp/terraform/blob/v1.9/CHANGELOG.md shows the upgrade guide at https://developer.hashicorp.com/terraform/language/v1.9.x/upgrade-guides
- No changes of concern to us on this one
- New interesting feature: "Input variable validation rules can refer to other objects" (cc @smerle33 for the modules validations)
[x] Bump Terraform version in packer-images
- [x] PR - https://github.com/jenkins-infra/packer-images/pull/1066
- [x] Release - https://github.com/jenkins-infra/packer-images/releases/tag/1.78.0
- [x] Deployment - https://github.com/jenkins-infra/kubernetes-management/pull/5408
[x] Ensure each projects works (run a local plan, if it fails, open a fix PR otherwise run a main build)
- [x] AWS - https://github.com/jenkins-infra/aws/pull/557
- [x] Azure - https://github.com/jenkins-infra/azure/pull/782
- [x] Azure Net - https://github.com/jenkins-infra/azure-net/pull/272
- [x] Cloudflare - https://github.com/jenkins-infra/cloudflare/pull/34
- [x] DigitalOcean - https://github.com/jenkins-infra/digitalocean/pull/203
- [x] Fastly - https://github.com/jenkins-infra/fastly/pull/110
- [x] Datadog - https://github.com/jenkins-infra/datadog/pull/253

Reproduction steps

No response

dduportal commented 3 months ago

Update:

Terraform changelogs looks good 👍
Terraform 1.9.1 deployed to infra.ci.jenkins.io
Starting batch PRs to bump projects versions

dduportal commented 3 months ago

Update:

All projects are now running with Terraform 1.9.1 with success for the "Plan" and "Ship" stages => we can safely operate the infrastructure
One last thing: the tests for the Azure project (and only this project) are now failing with the following (unclear) error:

11:01:31  Terraform has been successfully initialized!
11:01:31  Starting common-tests...
11:04:10  make: *** [Makefile:27: common-tests] Error 1
11:04:10  make: Leaving directory '/home/jenkins/agent/workspace/terraform-jobs_azure_main/.shared-tools/terraform'

=> Gotta check to understand what is wrong. Could be due to Terraform 1.9.1 or a mistake in https://github.com/jenkins-infra/helpdesk/issues/4166. First step would be to reproduce the error locally, check the line 27 of the Makefile and eventually reproduce with the terratest golang execution. If no progress, we can disable the test (as it is a flaky one if changing Terraform make it fail).

dduportal commented 3 months ago

Update:
* All projects are now running with Terraform 1.9.1 with success for the "Plan" and "Ship" stages => we can safely operate the infrastructure

* One last thing: the tests for the Azure project (and only this project) are now failing with the following (unclear) error:
11:01:31  Terraform has been successfully initialized!
11:01:31  Starting common-tests...
11:04:10  make: *** [Makefile:27: common-tests] Error 1
11:04:10  make: Leaving directory '/home/jenkins/agent/workspace/terraform-jobs_azure_main/.shared-tools/terraform'
=> Gotta check to understand what is wrong. Could be due to Terraform 1.9.1 or a mistake in #4166. First step would be to reproduce the error locally, check the line 27 of the Makefile and eventually reproduce with the terratest golang execution. If no progress, we can disable the test (as it is a flaky one if changing Terraform make it fail).

Retrieved the test logs (file ./common-tests.logs) and got the following error from it:

plan.go:17: 
                Error Trace:    plan.go:17
                                                        terraform_main_test.go:49
                Error:          Received unexpected error:
                                FatalError{Underlying: error while running command: exit status 1; ╷
                                │ Error: Invalid provider configuration
                                │ 
                                │ Provider "registry.terraform.io/hashicorp/azuread" requires explicit
                                │ configuration. Add a provider block to the root module and configure the
                                │ provider's required arguments as described in the provider documentation.
                                │ 
                                ╵
                                ╷
                                │ Error: unable to build authorizer: could not configure AzureCli Authorizer: obtaining subscription ID: obtaining account details: running Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.
                                │ 
                                │   with provider["registry.terraform.io/hashicorp/azuread"],
                                │   on <empty> line 0:
                                │   (source code not available)
                                │ 
                                ╵
                                ╷
                                │ Error: unable to build authorizer for Resource Manager API: could not configure AzureCli Authorizer: obtaining subscription ID: obtaining account details: running Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.
                                │ 
                                │   with provider["registry.terraform.io/hashicorp/azurerm"],
                                │   on providers.tf line 2, in provider "azurerm":
                                │    2: provider "azurerm" {
                                │ 
                                ╵
                                ╷
                                │ Error: unable to build authorizer for Resource Manager API: could not configure AzureCli Authorizer: obtaining subscription ID: obtaining account details: running Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.
                                │ 
                                │   with provider["registry.terraform.io/hashicorp/azurerm"].jenkins-sponsorship,
                                │   on providers.tf line 7, in provider "azurerm":
                                │    7: provider "azurerm" {
                                │ 
                                ╵}
                Test:           TestTerraformCluster

It's weird because azure-net does not fail with these errors 🤔

dduportal commented 3 months ago

Update:
* All projects are now running with Terraform 1.9.1 with success for the "Plan" and "Ship" stages => we can safely operate the infrastructure

* One last thing: the tests for the Azure project (and only this project) are now failing with the following (unclear) error:
11:01:31  Terraform has been successfully initialized!
11:01:31  Starting common-tests...
11:04:10  make: *** [Makefile:27: common-tests] Error 1
11:04:10  make: Leaving directory '/home/jenkins/agent/workspace/terraform-jobs_azure_main/.shared-tools/terraform'
=> Gotta check to understand what is wrong. Could be due to Terraform 1.9.1 or a mistake in #4166. First step would be to reproduce the error locally, check the line 27 of the Makefile and eventually reproduce with the terratest golang execution. If no progress, we can disable the test (as it is a flaky one if changing Terraform make it fail).

Retrieved the test logs (file ./common-tests.logs) and got the following error from it:

plan.go:17: 
                Error Trace:    plan.go:17
                                                        terraform_main_test.go:49
                Error:          Received unexpected error:
                                FatalError{Underlying: error while running command: exit status 1; ╷
                                │ Error: Invalid provider configuration
                                │ 
                                │ Provider "registry.terraform.io/hashicorp/azuread" requires explicit
                                │ configuration. Add a provider block to the root module and configure the
                                │ provider's required arguments as described in the provider documentation.
                                │ 
                                ╵
                                ╷
                                │ Error: unable to build authorizer: could not configure AzureCli Authorizer: obtaining subscription ID: obtaining account details: running Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.
                                │ 
                                │   with provider["registry.terraform.io/hashicorp/azuread"],
                                │   on <empty> line 0:
                                │   (source code not available)
                                │ 
                                ╵
                                ╷
                                │ Error: unable to build authorizer for Resource Manager API: could not configure AzureCli Authorizer: obtaining subscription ID: obtaining account details: running Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.
                                │ 
                                │   with provider["registry.terraform.io/hashicorp/azurerm"],
                                │   on providers.tf line 2, in provider "azurerm":
                                │    2: provider "azurerm" {
                                │ 
                                ╵
                                ╷
                                │ Error: unable to build authorizer for Resource Manager API: could not configure AzureCli Authorizer: obtaining subscription ID: obtaining account details: running Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.
                                │ 
                                │   with provider["registry.terraform.io/hashicorp/azurerm"].jenkins-sponsorship,
                                │   on providers.tf line 7, in provider "azurerm":
                                │    7: provider "azurerm" {
                                │ 
                                ╵}
                Test:           TestTerraformCluster

It's weird because azure-net does not fail with these errors 🤔

Found the issue! It was a mistake I made when updating the SOPS credentials as part of https://github.com/jenkins-infra/helpdesk/issues/4166:

As you can see, 2 fields of the Azure SP credential are empty in infra.ci.jenkins.io which explains the error.

Fixed by https://github.com/jenkins-infra/charts-secrets/commit/f41397342c738dffbd1c04a211624257e3718353
Applied with a run of kubernetes-management
Build is fixed as per https://github.com/jenkins-infra/azure/pull/785

dduportal commented 3 months ago

Terraform 1.9.1 is now used!

jenkins-infra / helpdesk