anishakj
commented
3 years ago @mogul Have you uninstalled zookeeper cluster, before uninstalling zookeeper operator. You can check by using kubectl get zk command
Open mogul opened 3 years ago
anishakj
commented
3 years ago @mogul Have you uninstalled zookeeper cluster, before uninstalling zookeeper operator. You can check by using kubectl get zk command
mogul
commented
3 years ago Yes:
We looked at the pre-delete hook and saw that it's checking for existing Zookeeper instances... We didn't create any while the chart was installed, and when we run the command from the hook we can confirm there are none:
anishakj
commented
3 years ago @mogul Could you please paste logs from pre-delete hook pod that gets created.?
SrishT
commented
3 years ago @mogul if the pre-delete hook is something do not need, you can easily disable it by setting hooks.delete to false while installing the zookeeper operator here
but in order to understand why the job is failing for you, we would need to see the logs within pre-delete hook pod that gets created.
anishakj
commented
3 years ago @mogul Could you please provide us logs if you are still seeing the issue or else can we close this?
anishakj
commented
3 years ago Closing this issue as there is no response from submitter. Please feel free to open the issue with logs, if the issue is seen again.
mogul
commented
3 years ago Hello, I'm once again hitting this problem now that the solr-operator requires zookeeper-operator 0.2.12.
@mogul if the pre-delete hook is something do not need, you can easily disable it by setting
hooks.deleteto false while installing the zookeeper operator here
This was enormously helpful, thanks! I'm able to use this setting to stay on 0.2.12 now despite the pre-delete hook problem.
but in order to understand why the job is failing for you, we would need to see the logs within pre-delete hook pod that gets created.
I tried to capture logs of the pre-delete pod, but the time between the job starting and the DeadlineExceeded message in the logs quoted above is just a few seconds:
client.go:268: [debug] Starting delete for "zookeeper-zookeeper-operator-pre-delete" Job
client.go:297: [debug] jobs.batch "zookeeper-zookeeper-operator-pre-delete" not found
client.go:122: [debug] creating 1 resource(s)
client.go:477: [debug] Watching for changes to Job zookeeper-zookeeper-operator-pre-delete with timeout of 5m0s
client.go:505: [debug] Add/Modify event for zookeeper-zookeeper-operator-pre-delete: ADDED
client.go:544: [debug] zookeeper-zookeeper-operator-pre-delete: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:505: [debug] Add/Modify event for zookeeper-zookeeper-operator-pre-delete: MODIFIED
client.go:268: [debug] Starting delete for "zookeeper-zookeeper-operator-pre-delete" Job
[...JUST A FEW SECONDS...]
Error: job failed: DeadlineExceeded
helm.go:81: [debug] job failed: DeadlineExceededThe pod is created and then gone again so fast that I'm not sure how to capture them... Is there some kubectl magic that would help with that? Or maybe the deadline is being expressed in the wrong magnitude units...?
anishakj
commented
3 years ago @mogul Could you please try collecting the logs by removing the the delete annotation from the job "helm.sh/hook-delete-policy": hook-succeeded, before-hook-creation, hook-failed
anishakj
commented
2 years ago @mogul Could you please update the logs.
Description
We used Helm to install the zookeeper-operator chart on Kubernetes 1.19. When we
helm uninstall zookeeperwe see...and the release is stuck in state "uninstalling":
Importance
(Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have))
blocker: We are trying to automate everything we do with terraform and this prevents us from being able to run
terraform destroywithout having to manually intervene to remove the release.Location
(Where is the piece of code, package, or document affected by this issue?)
This appears to be a result of the code introduced in https://github.com/pravega/zookeeper-operator/pull/301
When we try uninstalling with debugging on we see:
We looked at the pre-delete hook and saw that it's checking for existing Zookeeper instances... We didn't create any while the chart was installed, and when we run the command from the hook we can confirm there are none:
Suggestions for an improvement
(How do you suggest to fix or proceed with this issue?)
We can get around this manually for now by skipping the hooks during uninstall:
We can use the
disable_webhooksoption in the Terraform provider to get the same result, but that will skip all hooks (which is probably a bad thing to do... not sure what other hooks the chart has in it).For our current situation the best workaround is to use the previous version of the chart, but we'd rather not miss out on future improvements, so we're hoping to see this fixed.