rover deploy fails with "Error: Saved plan is stale"

[sschne]

When calling rover deploy apply with multiple landing zones in the symphony.yml, the deployment fails at the 2nd landing zone with "Error: Saved plan is stale". It looks like as the plan file gets carried on, so the 2nd landingzone terraform command is invoked with the same plan file as the first as seen in the output below.

Command: rover --symphony-config /tf/caf/symphony.yml -level all --environment demo deploy apply symphony.yml:

levels:
  - level: level0
    stacks:
      - stack: launchpad
        repo: base
        landingZonePath: /tf/caf/landingzones/caf_launchpad
        configurationPath: /tf/caf/configuration/level0/launchpad
        launchpad: true
        tfState: caf_launchpad.tfstate
  - level: level1
    stacks:
      - stack: enterprise_scale
        repo: base
        landingZonePath: /tf/caf/landingzones/caf_solution/add-ons/caf_eslz
        configurationPath: /tf/caf/configuration/level1/enterprise_scale
        tfState: caf_eslz.tfstate

Output:

@calling deploy_landingzone
Deploying '//tf/caf/landingzones/caf_solution/add-ons/caf_eslz'
Terraform version 0.15 or greater
Upgrading modules...
Downloading Azure/caf-enterprise-scale/azurerm 0.3.3 for enterprise_scale...
- enterprise_scale in /__w/_temp/rover/modules/enterprise_scale
- enterprise_scale.management_group_archetypes in /__w/_temp/rover/modules/enterprise_scale/modules/archetypes
- enterprise_scale.management_resources in /__w/_temp/rover/modules/enterprise_scale/modules/management

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding hashicorp/time versions matching ">= 0.7.0"...
- Finding hashicorp/azurerm versions matching ">= 2.41.0, ~> 2.65.0"...
- Using hashicorp/time v0.7.2 from the shared cache directory
- Installing hashicorp/azurerm v2.65.0...
- Installed hashicorp/azurerm v2.65.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│ 
│   on main.tf line 10, in terraform:
│   10:   experiments      = [module_variable_optional_attrs]
│ 
│ Experimental features are subject to breaking changes in future minor or
│ patch releases, based on feedback.
│ 
│ If you have feedback on the design of this feature, please open a GitHub
│ issue to discuss it.
╵

Terraform has been successfully initialized!
Terraform init return code 0
calling apply
@calling apply
running terraform apply
Running Terraform apply with plan /__w/_temp/rover/tfstates/level0/tfstate/caf_launchpad.tfplan
Terraform version 0.15 or greater
╷
│ Error: Saved plan is stale
│ 
│ The given plan file can no longer be applied because the state was changed
│ by another operation after the plan was created.
╵
╷
│ Error: Could not load plugin
│ 
│ 
│ Plugin reinitialization required. Please run "terraform init".
│ 
│ Plugins are external binaries that Terraform uses to access and manipulate
│ resources. The configuration provided requires plugins which can't be
│ located,
│ don't satisfy the version constraints, or are otherwise incompatible.
│ 
│ Terraform automatically discovers provider requirements from your
│ configuration, including providers used in child modules. To see the
│ requirements and constraints, run "terraform providers".
│ 
│ 7 problems:
│ 
│ - failed to instantiate provider "registry.terraform.io/aztfmod/azurecaf"
│ to obtain schema: unknown provider "registry.terraform.io/aztfmod/azurecaf"
│ - failed to instantiate provider "registry.terraform.io/hashicorp/azuread"
│ to obtain schema: unknown provider
│ "registry.terraform.io/hashicorp/azuread"
│ - failed to instantiate provider "registry.terraform.io/hashicorp/external"
│ to obtain schema: unknown provider
│ "registry.terraform.io/hashicorp/external"
│ - failed to instantiate provider "registry.terraform.io/hashicorp/null" to
│ obtain schema: unknown provider "registry.terraform.io/hashicorp/null"
│ - failed to instantiate provider "registry.terraform.io/hashicorp/random"
│ to obtain schema: unknown provider "registry.terraform.io/hashicorp/random"
│ - failed to instantiate provider "registry.terraform.io/hashicorp/template"
│ to obtain schema: unknown provider
│ "registry.terraform.io/hashicorp/template"
│ - failed to instantiate provider "registry.terraform.io/hashicorp/tls" to
│ obtain schema: unknown provider "registry.terraform.io/hashicorp/tls"
│ 
╵
Terraform apply return code: 1

[arnaudlh]

Closing as merged in 2112.