Data sources are not refreshed prior to destroy (terraform 0.14.x backport)

[dak1n1]

This issue has already been fixed, but I wanted to request a backport to 0.14. The problem is that data sources are not being refreshed prior to destroy. Here's the PR with the fix, which was released with 0.15-beta1. I did my testing using 0.15-beta1 and it seems to work. https://github.com/hashicorp/terraform/pull/27408

If I understand correctly, there was a separate refresh step that was done in 0.13 which allowed data sources to be read prior to destroy. In 0.14, the separate refresh step was removed, but PR #27408 enables the behavior again.

Terraform Version

0.14.7

Terraform Configuration Files

Please let me know if you end up needing a reproducer and I'll create one. I figured since it's already been solved, this is ok to skip this time. Here's approximately what I'm working with:

data "aws_eks_cluster" "default" {
  name = module.cluster.cluster_id
}

data "aws_eks_cluster_auth" "default" {
  name = module.cluster.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.default.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.default.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.default.token
}

module "cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "14.0.0"

  cluster_name     = "test"
  cluster_version  = "1.18"
  manage_aws_auth  = true ### this uses the kubernetes provider

  workers_group_defaults = {
    root_volume_type = "gp2"
  }
}

Debug Output

Crash Output

Expected Behavior

terraform destroy should destroy the resources that have been created.

Actual Behavior

The data sources that contain the Kubernetes cluster credentials are not read prior to destroy. So the Kubernetes provider initializes using an empty config, and resources are not deleted.

Steps to Reproduce

1. Try to build an EKS cluster using the above config. The module contains the Kubernetes provider and will provision a resource that triggers this issue. (Use terraform 0.14.x for this).
2. Wait 15 minutes for the token to expire.
3. terraform destroy

Additional Context

Some users are avoiding terraform 0.14 because of this bug. It impacts anyone who is creating a Kubernetes cluster (GKE, EKS, AKS, etc) and then placing Kubernetes or Helm provider resources on top of that cluster. While we discourage keeping the cluster infrastructure in the same Terraform State as the dependent resources, it is a very common workflow, so I wanted to help them out if possible.

References

https://github.com/terraform-aws-modules/terraform-aws-eks/pull/1230 https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1162

[jbardin]

Thanks @dak1n1,

I understand the concern, but the backport was avoided in this case because it is a bit more intrusive that we would like for a patch release, and since the changes do not apply to the v0.14 codebase, it would require additional work and risk to recreate the same behavior. The 0.15 release is forthcoming, and should be a relatively smooth upgrade.

Are there cases where the workaround of using refresh before destroy does not accomplish the same result?

[dak1n1]

Thanks for explaining, that makes sense. And yes, terraform refresh works around the issue. Users generally hit this error and may not know a refresh is needed:

Error: Delete "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp [::1]:80: connect: connection refused
Error: Unauthorized

I'll keep looking for ways to improve this user experience since the Kubernetes provider is the one giving this error. You can close this issue, thanks!

[jbardin]

Thanks for the update @dak1n1!

[rmasclef]

Hi, I can give an example where a patch might help a lot 👍

As a user of TerraformCloud using VCS integration*, I can not perform a terraform refresh or select a beta version of the terraform binary to be used for a given workspace ... therefore I'm stuck waiting for the v0.15.0 to be out 😢

*i.e not using terraform cli from my CI or my PC

This is not a huge issue as I can still perform a refresh from my laptop but it is a bit out of my terraform workflow 🤷‍♂️ (i.e I can not see on tfCloud that I performed a plan and an apply using a specific version from my PC)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.