Unable to delete workspace when Apply fails

[rojopolis]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

terraform-k8s & Kubernetes Version

AWS EKS 1.14 Terraform 0.12.24 Operator: hashicorp/terraform-k8s:0.1.1-alpha

Affected Resource(s)

Workspace

Terraform Configuration Files

# Copy-paste your Terraform configuration from the operator here.
# To retrieve the configuration, use `kubectl -n $NAMESPACE describe configmap $WORKSPACE_NAME`
terraform {
           backend "remote" {
             organization = "parkside_securities"

             workspaces {
               name = "default-buckets"
             }
           }
         }
         variable "bucket" {}
         output "buckets" {
           value = module.operator.buckets
         }
         module "operator" {
           source = "git@github.com:rojopolis/terraform-aws-s3-bucket.git"
           bucket = var.bucket
         }

Debug Output

Expected Behavior

kubectl delete workspace/buckets

Actual Behavior

kubectl hangs (forever?)

sync-workspace logs:

{"level":"error","ts":1586469753.5856676,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"workspace-controller","request":"default/buckets","error":"destroy had error: <nil>","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v │
│ 0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runn │
│ er/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.Ji │
│ tterUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.U │
│ ntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:88"}

Steps to Reproduce

1. Create a Workspace in K8s that can't be applied successfully.
2. Try to delete the Workspace resource.

Important Factoids

My config was invalid because the module in use requires a list as an input variable #11 . It seems like the transition of failed->deleted should be valid.

References

11

• 0000

[joatmon08]

Thanks for reporting! Just to clarify, would there ever be a concern with deleting the workspace and stranding the resources created by the provider (e.g., resources created in AWS no longer in state due to workspace deletion)? The current workflow does not delete the workspace because the finalizer is written to accommodate for this concern. I think we could potentially handle this by checking for whether or not there are resources in state but I wanted to get some thoughts on the approach.

[rojopolis]

Just to clarify, would there ever be a concern with deleting the workspace and stranding the resources created by the provider (e.g., resources created in AWS no longer in state due to workspace deletion)?

Yes, That seems like it would be a problem. In my trivial example that case isn't possible, but if resources are created they should be destroyed.

[joshvermast]

FYI this also happens when the Destroy Plan fails in Terraform Cloud. The workspace is literally undeletable and stuck.