Terraspace init ignores required_providers for dependent stacks

pilchkinstein-perspectum commented 2 years ago

Checklist

[ ] Upgrade Terraspace: Are you using the latest version of Terraspace? This allows Terraspace to fix issues fast. There's an Upgrading Guide: https://terraspace.cloud/docs/misc/upgrading/
[x] Reproducibility: Are you reporting a bug others will be able to reproduce and not asking a question. If you're unsure or want to ask a question, do so on https://community.boltops.com
[x] Code sample: Have you put together a code sample to reproduce the issue and make it available? Code samples help speed up fixes dramatically. If it's an easily reproducible issue, then code samples are not needed. If you're unsure, please include a code sample.

My Environment

Software	Version
Operating System	Ubuntu 20.04 LTS
Terraform	1.1.3
Terraspace	1.0.6
Ruby	3.0.3

Expected Behaviour

Terraspace should allow different stacks to use different provider versions

Current Behavior

Terraspace downloads the latest version of a provider and then fails during the plan/up/down stage with an error such as

│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider
│ hashicorp/azuread: locked provider registry.terraform.io/hashicorp/azuread
│ 2.19.1 does not match configured version constraint >= 2.7.0, 2.7.0, ~>
Terraform reinitialization required detected. Will run `terraform init` and try again.

The error we observe appears to be related to a 3-tier dependency and sourcing remote terraform modules. While trying to replicate the issue for this report I could only demonstrate that the providers downloaded during the Downloading tfstate files for dependencies defined in tfvars... stage were not the ones declared within the stack itself. This did not cause the version constraint error above. I cannot rule out that it is our Terraform module which tips an error which Terraspace proceeds through into the fatal I've listed above

Step-by-step reproduction instructions

terraspace new stack A
terraspace new stack B
terraspace new stack C
TS_ENV=base terraspace seed B
TS_ENV=base terraspace seed C
Set the contents of each file as follows:

A/main.tf

# A - this provides an output that B will depend on
terraform {
  required_providers {
    azuread = {
      source  = "hashicorp/azuread"
      # Not latest (which is currently 2.19.1) but newer than stack B
      version = "~>2.18.0"
    }
  }
}

output "id" {
  value = "11111111111111111"
}

B/main.tf

# B - this depends on an output that A will provide
terraform {
  required_providers {
    azuread = {
      source  = "hashicorp/azuread"
      # Older version than stack A
      version = "~>2.7.0"
    }
  }
}

variable "id" { }

locals {
  id = var.id
}

output "id" {
  value = local.id
}

B/tfvars/base.tfvars

id = <%= output('A.id') %>

C/main.tf

# C - this depends on an output that B will provide

# No provider versions specified
//terraform {
//  required_providers { }
//}

variable "id" { }

C/tfvars/base.tfvars

id = <%= output('B.id') %>

terraspace up A
terraspace up B
terraspace clean cache
terraspace plan B

At this point part of the problem is observed The log for B.log shows an init step due to the presence of the child dependency C. Below is the start of the log from our stack which annoyingly I cannot fully reproduce. It's possible to see the 4 lines where the latest version of plugins are installed:-

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding latest version of hashicorp/random...
- Finding latest version of hashicorp/template...
- Finding latest version of hashicorp/aws...
- Finding latest version of hashicorp/azuread...
- Installing hashicorp/random v3.1.2...
- Installed hashicorp/random v3.1.2 (signed by HashiCorp)
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing hashicorp/aws v4.8.0...
- Installed hashicorp/aws v4.8.0 (signed by HashiCorp)
- Installing hashicorp/azuread v2.19.1...
- Installed hashicorp/azuread v2.19.1 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
...

This shows that something is causing Terraspace to go and download the latest providers for aws, azuread etc - not the ones specified in the providers in either A or B

Code Sample

See above

Solution Suggestion

I propose that terraspace run terraform init within each stack directory (and obey stack specific required_provider constraints) before moving on to the "Downloading tfstate files for dependencies defined in tfvars..." step

pilchkinstein-perspectum commented 2 years ago

@tongueroo I've spent about 24hrs trying to isolate and describe the steps to reproduce this issue. It is not intermittent on our codebase but I'm afraid I cant share all of that with you. I think I've demonstrated part of the problem (an error due to incorrect versions being selected) but I cant demonstrate the fatal that we see at our place.

pilchkinstein-perspectum commented 2 years ago

After upgrading to Terraspace 1.1.7 we dont see this issue

boltops-tools / terraspace