Terraform Cloud: Multiple 'schema: unknown provider "registry.terraform.io/-/..."' errors with official providers when upgrading to 0.13

jnewland commented 4 years ago

Terraform Version

Terraform v0.13.1
...
+ provider registry.terraform.io/-/random v2.3.0
...
+ provider registry.terraform.io/hashicorp/random v2.3.0

Terraform Configuration Files

terraform {
  backend "remote" {
  ...
  }
  required_version = "~> 0.13.1"
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 2.1"
    }
  ...
  }

Debug Output

Unavailable, using terraform cloud

Expected Behavior

The first apply on a new version should not fail after following the upgrade guide and obtaining a clean plan on the new version.

Actual Behavior / Reproduction steps

Read upgrade guide
Take all actions indicated:
- Run terraform 0.13upgrade
- Update provider syntax
Run terraform init locally (returns cleanly)
Obtain clean plan using terraform cloud
Merge PR
Attempt to apply using terraform cloud
Encounter this error:

Terraform v0.13.1
Initializing plugins and modules...

Error: Could not load plugin

Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

5 problems:

- Failed to instantiate provider "registry.terraform.io/-/aws" to obtain
schema: unknown provider "registry.terraform.io/-/aws"
- Failed to instantiate provider "registry.terraform.io/-/local" to obtain
schema: unknown provider "registry.terraform.io/-/local"
- Failed to instantiate provider "registry.terraform.io/-/mongodbatlas" to
obtain schema: unknown provider "registry.terraform.io/-/mongodbatlas"
- Failed to instantiate provider "registry.terraform.io/-/random" to obtain
schema: unknown provider "registry.terraform.io/-/random"
- Failed to instantiate provider "registry.terraform.io/-/template" to obtain
schema: unknown provider "registry.terraform.io/-/template"

Additional Context

https://github.com/hashicorp/terraform/issues/25705 indicates that this is an expected error, which is confusing to me after reading the upgrade guide both before and after encountering it:

The error message asks me to run terraform init, which is impossible in my workflow
Remediation steps for similar errors are included in an in-house providers section: we are not using any in-house providers
The FAQ entry for this point indicates that these messages should not cause workflow problems:

While this does not cause any problems for Terraform, it has been confusing.

Is it expected that users that do not use any in-house providers encounter this error? Are the remediation steps the same? I'd like to confirm before taking any destructive actions like terraform state replace-provider. Thanks in advance!

References

/cc https://github.com/hashicorp/terraform/issues/25705

caronicolas commented 4 years ago

I have the same one yesterday, I use the command "terraform state replace-provider" and then it works.

gdurandvadas commented 4 years ago

I'm having this exact problem but I just tried with one of my workspaces and passed. Could be they rolled-out a fix?

jnewland commented 4 years ago

I have the same one yesterday, I use the command "terraform state replace-provider" and then it works.

I expect that this will be the case, but I'd like to double check with someone from Hashicorp before doing so since the upgrade guide seems to specifically say that I shouldn't be experiencing any workflow problems.

I'm having this exact problem but I just tried with one of my workspaces and passed. Could be they rolled-out a fix?

I just retried my run and it failed in the exact same fashion. 😢

gdurandvadas commented 4 years ago

Ok, I figured why it worked. I ran those workspaces locally first and then on Terraform Cloud. The other option is what @caronicolas mentioned, here is an example for the aws provider:

➜ terraform state replace-provider 'registry.terraform.io/-/aws' 'registry.terraform.io/hashicorp/aws'
Acquiring state lock. This may take a few moments...
Terraform will perform the following actions:

  ~ Updating provider:
    - registry.terraform.io/-/aws
    + registry.terraform.io/hashicorp/aws

Changing X resources:

  {edited}

Do you want to make these changes?
Only 'yes' will be accepted to continue.

Enter a value: yes

Successfully replaced provider for X resources.

If you see the error message, it says that can't find the provider in "registry.terraform.io/-", so replacing that with the correct HashiCorp URL, fixes the issue.

alisdair commented 4 years ago

Thanks for reporting this issue! I don't think you should be seeing this error.

Unfortunately, I'm unable to reproduce the problem given your steps. Here's what I did, all commands executed locally using the remote backend:

Created a new Terraform configuration:

terraform {
  backend "remote" {
    workspaces {
      name = "26075"
      organization = "<redacted>"
    }
  }
}

resource "random_pet" "friend" {
}

Run terraform-0.12.28 init: successfully creates Terraform Cloud workspace
Run terraform-0.12.28 apply: applies, creates resource, saves state
Run terraform-0.13.1 0.13upgrade: creates versions.tf file as expected
Update my Terraform Cloud workspace to use version 0.13.1 (using the UI)
Run terraform-0.13.1 plan: speculative plan succeeds on Terraform Cloud
Run terraform-0.13.1 apply: applies on Terraform Cloud, no changes
Changed the configuration (set length = 3 on the resource), reran apply: still works

Using these steps as a starting point, are you able to find a way to reproduce the problem? Without a simple reproduction test case, it's very difficult to figure out what could be going wrong here.

jnewland commented 4 years ago

Hey @alisdair! I'm unable to reproduce using those steps. I also tried the same set of steps with a versions.tf containing the following in the 0.12.28 phase:

provider "random" {
   version = "~> 2.1"
 }

That didn't reproduce the issue either.

I'm inclined to move forward with the terraform replace-provider remediation steps outlined above to unblock my environment given others experience here. But before I do, I'm going to leave behind some debugging notes that may help you or others figure out what's going on here.

terraform providers outputs the following information at the end, which seems to be a sign of the problem I'm experiencing.

Providers required by state:

    provider[registry.terraform.io/-/local]

    provider[registry.terraform.io/-/mongodbatlas]

    provider[registry.terraform.io/-/null]

    provider[registry.terraform.io/-/random]

    provider[registry.terraform.io/-/template]

    provider[registry.terraform.io/-/aws]

    provider[registry.terraform.io/-/helm]

    provider[registry.terraform.io/-/kubernetes]

terraform init also warns me that I don't have version constraints set for the un-namepaced name of the same set of modules when run locally, even though I have a full required_providers section with version constraints for each:

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* -/aws: version = "~> 3.4.0"
* -/helm: version = "~> 1.2.4"
* -/kubernetes: version = "~> 1.12.0"
* -/local: version = "~> 1.4.0"
* -/mongodbatlas: version = "~> 0.6.3"
* -/null: version = "~> 2.1.2"
* -/random: version = "~> 2.3.0"
* -/template: version = "~> 2.1.2"

In addition to those explicit dependencies, my configuration also includes resources created by several modules that use the following backward-compatible form to declare provider dependencies:

https://github.com/terraform-aws-modules/terraform-aws-s3-bucket/blob/master/versions.tf

terraform {
  required_version = ">= 0.12.6, < 0.14"

  required_providers {
    aws = ">= 3.0, < 4.0"
  }
}

jnewland commented 4 years ago

Hi folks! I was able to resolve most of this error by running the terraform state replace-provider commands above. Unfortunately, I'm now stuck with the following error that I can't get myself out of with terraform state replace-provider:

Terraform v0.13.2
Initializing plugins and modules...

Error: Could not load plugin

Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

2 problems:

- Failed to instantiate provider "registry.terraform.io/-/helm" to obtain
schema: unknown provider "registry.terraform.io/-/helm"
- Failed to instantiate provider "registry.terraform.io/-/kubernetes" to
obtain schema: unknown provider "registry.terraform.io/-/kubernetes"

Here's my configuration for the two providers in question:

terraform {
  backend "remote" {
    organization = "[redacted]"
    workspaces {
      name = "[redacted]"
    }
  }
  required_version = "~> 0.13.1"
    helm = {
      source  = "hashicorp/helm"
      version = "~> 1.2"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 1.11.2"
    }
   # ...
  }
}

provider "kubernetes" {
  load_config_file = false

  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes {
    load_config_file = false

    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

replace-provider fails like so:

terraform state replace-provider -- registry.terraform.io/-/helm registry.terraform.io/hashicorp/helm              

No matching resources found.

Any suggestions?

jnewland commented 4 years ago

Ahh, please ignore. This error was due to a mismatch between my local CLI environment and terraform cloud. Closing since the original issue has been resolved.

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform