unknown provider "registry.terraform.io/-/stateful" (we don't use this)

[antonysouthworth-halter]

We encounter the following error when using plan_only: true:

2021/12/22 00:05:10 Plan Error: Failed to upload plan file to TF backend. Use `fly intercept` to SSH into this container and view /tmp/tf-plan.log for more logs. Error: Failed to run Terraform command: exit status 1

Contents of that log file (click to expand)

``` bash-5.1# cat /tmp/tf-plan.log ╷ │ Warning: Value for undeclared variable │ │ The root module does not declare a variable named │ "discovery_component_secret" but a value was found in file │ "/tmp/terraform-resource-out2308508115/2248223420vars-file.tfvars.json". If │ you meant to use this value, add a "variable" block to the configuration. │ │ To silence these warnings, use TF_VAR_... environment variables to provide │ certain "global" settings to all configurations in your organization. To │ reduce the verbosity of these warnings, use the -compact-warnings option. ╵ ╷ │ Warning: Value for undeclared variable │ │ The root module does not declare a variable named "api_version" but a value │ was found in file │ "/tmp/terraform-resource-out2308508115/4206111767.tfvars". If you meant to │ use this value, add a "variable" block to the configuration. │ │ To silence these warnings, use TF_VAR_... environment variables to provide │ certain "global" settings to all configurations in your organization. To │ reduce the verbosity of these warnings, use the -compact-warnings option. ╵ ╷ │ Warning: Values for undeclared variables │ │ In addition to the other similar warnings shown, 2 other variable(s) │ defined without being declared. ╵ ╷ │ Error: Could not load plugin │ │ │ Plugin reinitialization required. Please run "terraform init". │ │ Plugins are external binaries that Terraform uses to access and manipulate │ resources. The configuration provided requires plugins which can't be │ located, │ don't satisfy the version constraints, or are otherwise incompatible. │ │ Terraform automatically discovers provider requirements from your │ configuration, including providers used in child modules. To see the │ requirements and constraints, run "terraform providers". │ │ failed to instantiate provider "registry.terraform.io/-/stateful" to obtain │ schema: unknown provider "registry.terraform.io/-/stateful" │ ```

but when I run terraform providers, no such plugin is found:

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/random]
├── provider[registry.terraform.io/hashicorp/template]
├── provider[registry.terraform.io/hashicorp/aws]
└── module.lambda_auth
    └── provider[registry.terraform.io/hashicorp/aws]

Providers required by state:

    provider[registry.terraform.io/hashicorp/aws]

    provider[registry.terraform.io/hashicorp/random]

Extra weird: we run terraform plan against our dev, staging, and prod environments, but this issue only happens in staging and prod...

Please let me know if any other details would be helpful

[antonysouthworth-halter]

ok; looks like the plan files for staging and prod are using 0.12-style provider names in the state files they are trying to upload; snippet of diff dev-plan.json staging-plan.json

42c41
<       "provider": "provider[\"github.com/ashald/stateful\"]",
---
>       "provider": "provider.stateful",
47,49c46,48

is it safe to delete the plan state files? Presumably they would get re-created anyways with the new provider styles.

[antonysouthworth-halter]

I did it anyway; delete the plan files in S3 (we are using S3 remote backend) and delete the entries in Dynamo lock table, then re-running plan and it succeeded. Will close this since it's fixed, but would love to hear maintainer's ideas about how we could have ended up with 0.12 plan files which only now are breaking (an earlier test, ~Nov 2, same repo but no such error; seems weird to me that it's come up now).