Confusing error message when missing a proxy provider configuration

[bcsgh]

Terraform Version

$ terraform version
Terraform v0.13.4
+ provider registry.terraform.io/hashicorp/aws v3.11.0

Terraform Configuration Files

Initial state

$ find -name *.tf | more $(cat) | cat
::::::::::::::
./main.tf
::::::::::::::
variable "aws_access_key" {}

variable "aws_secret_key" {}

provider "aws" {
  access_key = var.aws_access_key
  secret_key = var.aws_secret_key
  region = "us-west-2"
}

module "m" {
  source = "./m"
}
::::::::::::::
./m/main.tf
::::::::::::::
provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}

data "aws_sns_topic" "x" {
  provider = aws.us_east_1
  name = "TLSCertHealthChecks"
}

resource "aws_s3_bucket_object" "X" {
  provider = aws.us_east_1
  bucket = "my-us-east-1-bucket"
  key = "foo"
  content = data.aws_sns_topic.x.arn
}

Run terraform init ; terraform apply (which works) Modify the config to:

$ find -name *.tf | more $(cat) | cat
::::::::::::::
./main.tf
::::::::::::::
variable "aws_access_key" {}

variable "aws_secret_key" {}

provider "aws" {
  access_key = var.aws_access_key
  secret_key = var.aws_secret_key
  region = "us-west-2"
}

provider "aws" {    // ADD
  alias  = "us_east_1"
  region = "us-east-1"
}

module "m" {
  source = "./m"
  providers = {    // ADD
    aws.us_east_1 = aws.us_east_1
  }
}
::::::::::::::
./m/main.tf
::::::::::::::
// rm aws.us_east_1
data "aws_sns_topic" "x" {
  provider = aws.us_east_1
  name = "thing"
}

resource "aws_s3_bucket_object" "X" {
  provider = aws.us_east_1
  bucket = "my-us-east-1-bucket"
  key = "foo"
  content = data.aws_sns_topic.x.arn
}

Try to reapply and get this:

$ terraform apply

Error: Provider configuration not present

To work with module.m.aws_s3_bucket_object.X its original provider
configuration at
module.m.provider["registry.terraform.io/hashicorp/aws"].us_east_1 is
required, but it has been removed. This occurs when a provider configuration
is removed while objects created by that provider still exist in the state.
Re-add the provider configuration to destroy module.m.aws_s3_bucket_object.X,
after which you can remove the provider configuration again.

Ok, but destroying the object is not an option, it's still in use (true in the non-toy case).

Workaround: terraform state rm module.m.aws_s3_bucket_object.X and then re-import the object:

$ terraform state rm module.m.aws_s3_bucket_object.X
Removed module.m.aws_s3_bucket_object.X
Successfully removed 1 resource instance(s).
$ terraform state rm module.m.data.aws_sns_topic.health   ### Just to be safe
Removed module.m.data.aws_sns_topic.health
Successfully removed 1 resource instance(s).
$ cat terraform.tfstate  ### Check that terraform knows nothing about anything!
{
  "version": 4,
  "terraform_version": "0.13.4",
  "serial": 8,
  "lineage": "0e6ecb36-fdbd-03e6-0829-d9de9a51b207",
  "outputs": {},
  "resources": []
}
$ terraform import module.m.aws_s3_bucket_object.X foo

Error: Provider configuration not present

To work with module.m.aws_s3_bucket_object.X its original provider
configuration at
module.m.provider["registry.terraform.io/hashicorp/aws"].us_east_1 is
required, but it has been removed. This occurs when a provider configuration
is removed while objects created by that provider still exist in the state.
Re-add the provider configuration to destroy module.m.aws_s3_bucket_object.X,
after which you can remove the provider configuration again.

Error: Provider configuration not present

To work with module.m.aws_s3_bucket_object.X (import id
"delete_me/benjamin.shropshire") its original provider configuration at
module.m.provider["registry.terraform.io/hashicorp/aws"].us_east_1 is
required, but it has been removed. This occurs when a provider configuration
is removed while objects created by that provider still exist in the state.
Re-add the provider configuration to destroy module.m.aws_s3_bucket_object.X
(import id "foo"), after which you can remove the
provider configuration again.

Um... WAHT?!

How doe sit even know that exists!?

From here I've tried, renaming the resource, renaming the provider pointing the provider at a different region, making the resource refer to a different object, rm -rf .terraform, rm -rf ~/.terraform.d pkill -9 $NAMES_OF_PROVIDERS and even rebooting.

I've even (in the real-world application, for the cases where I could) terraform destroyed the resources before removing the provider. When I try to edit the config of those resource (which are not associated with any resource) trigger similar errors.

I don't see any way to move forward from here.

Expected Behavior

Moving from the discouraged setup of "providers in sub modules" to the encouraged setup of "all providers at the root config" should be viable, preferably without destroying the resources.

Actual Behavior

Terraform seems to have some hidden state somewhere that is tracking which provider was used to create resource and then refusing to allow any modification to it regardless of anything.

[bcsgh]

And I'm not the only one who has run into this sort of thing: https://discuss.hashicorp.com/t/refactoring-to-move-a-provider-from-a-child-module-from-the-root-module/10954

[apparentlymart]

Hi @bcsgh,

I think what's going on here is that your child module doesn't have a proxy provider configuration to indicate that your module will be passed an aliased provider named us_east_1, and so Terraform is getting confused.

If you add a proxy configuration to your child module then I think this should work as you intended:

provider "aws" {
  alias = "us_east_2"

  # No other arguments, because this is just a proxy that
  # will be associated with the parent module's provider
  # configuration.
}

Terraform's handling of provider configurations in child modules is unfortunately rather messy because of the efforts it goes to in order to try to remain compatible with the practice of putting provider configurations in modules from Terraform 0.10 and earlier. We are hoping to improve this eventually, but we suspect that doing so will require at least partially breaking compatibility with the older practice, and so we've not attempted to tackle it yet.

I hope the proxy configuration insight above helps you move forward right now. For the purpose of this issue we'll use it to represent that Terraform didn't give good feedback here. It seems like we could perhaps make Terraform produce a special error message in this specific situation (where there's no proxy provider configuration to match with a provider configuration explicitly assigned in the calling module block), which can hopefully serve as a modest improvement in the short term, rather than waiting for the longer-term design effort.

[bcsgh]

Better error messages would be a big improvement, regardless of what the situation turns out to be.

To check if I'm understanding you correctly, would a good message say something like:

Resource foo depends on provider bar which could not be found.
It may have been removed from the module, in which case it would
need to be restored in order to manipulate foo, or it may have
been moved into a different module, in which case a proxy provider
needed to be added in it's place.  

That said, and I'm working just from memory right now, I think I got the same issue with a proxy provider added. Though that might have been a result of having a un-aliased, default(?) provider (which it seems you can't create a proxy for) which was already inherited from the root. It's not clear what the expected approach for that case is.

[bcsgh]

FWIW: adding a proxy provider seems to have gotten thing un-stuck. So it seems the bug here is an amusingly misleading error message.

[apparentlymart]

Thanks for confirming, @bcsgh. With that in mind, I've relabeled this issue and changed its title to reflect the focus on improving the error message for now.

Hopefully some bigger changes around how provider passing between modules works will come later, but improving the error message here should ideally not block on the bigger design work.

[bcsgh]

Thanks!

IMHO, improving error messages is almost always worth doing, even for temporary corner cases. (After all, nothing is as temporary as we hope it will be.)

Random idea: add a flag that causes each error message to kick out a link to a distinct wiki page where more details & examples can be provide and user experience can be accumulated?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.