search

kubevela / terraform-controller

A Kubernetes Terraform Controller, managing Cloud resources through Kubernetes
Apache License 2.0
155 stars 68 forks source link

Deleting a configuration causes kubectl to hang indefinitely #83

Open mattkirby opened 3 years ago

mattkirby commented 3 years ago

When destroying a configuration resource kubectl will hang indefinitely.

With a configuration called 'myconfig' run kubectl delete configuration myconfig. 'configuration.terraform.core.oam.dev "myconfig" deleted' is returned and then kubectl hangs, presumably due to the finalizer 'configuration.finalizers.terraform-controller'. Even if I wait for the destroy job associated with the configuration to complete, the kubectl command for deleting the resource never returns. The same issue occurs whether the destroy job succeeds, or fails.

Workaround: To get the configuration to be removed the only solution I have found requires editing the configuration resource manually to remove the finalizer, replacing it with null.

Kubectl version is v1.20.8 on client and server.

The issue has been reproduced on both v0.1.16 and master.

zzxwill commented 3 years ago

@mattkirby When there is a syntax error in the Configuration, like incomplete of required_provider, terraform destroy will fail to complete when you kubectl delete configuration myconfig. In this situation, we should detect it's an error of Terraform HCL syntax and we can delete the configuration directly, without checking whether the cloud resource is deleted of not.

Are you interested to make a PR for it?

mattkirby commented 3 years ago

I find that deleting the configuration hangs indefinitely whether the destroy job is completed successfully, or not. With a successful destroy, and job marked as complete, in my experience it still fails to finish deleting the configuration. I have reproduced this when there are errors with the apply/destroy job, and when both complete without issue, so it seems like the behavior is different than I expect. I have also seen the issue with both GCP and EC providers. Do you find you are able to delete a configuration successfully without kubectl hanging indefinitely?