search

vertica / vertica-kubernetes

Operator, container and Helm chart to deploy Vertica in Kubernetes
Apache License 2.0
44 stars 25 forks source link

Graceful Vertica node stop in case of pod termination #935

Open nyikesda opened 3 weeks ago

nyikesda commented 3 weeks ago

PreStop hook in the container spec is a lifecycle hook that allows you to execute a command in a container right before it is terminated. This can be useful for performing cleanup or graceful shutdown tasks before a pod is taken down.

What do you think, would not be a good setting in the container in preStop hook to stop the vertica node gracefully?

Actually, we cannot set such kind of thing, because it cannot be configured in the operator configuration.

Kubernetes start pod termination in several cases (for example: in case of pod eviction because of resources, or kubernetes node drain) and in these cases the operator does not notice to stop the kubernetes on that pod gracefully.

roypaulin commented 3 weeks ago

Hi @nyikesda. I get the idea. preStop hooks are indeed very useful in many cases. They are also to be used carefully as you would need to set a `terminationGracePeriodSeconds long enough to allow the preStop hook to complete but is not excessively long I will look into what kind of actions can be executed through preStop hook and their benefits on Vertica.

Do you mind telling me an example of specific action you would like to set in the preStop hook?

nyikesda commented 3 weeks ago

I know that admintool usage is deleted from operator 2.0.0, but with admintool I would run a script which would call: /opt/vertica/bin/adminTools -t stop_node --hosts v_vertica_db_node0001 in case of node1 to stop the node gracefully when a pod is terminated.

I hope that this idea would be possible in case of vcluster-ops as well with curl commands for example.

roypaulin commented 2 weeks ago

We are going to investigate adding a preStop hook to the vertica pods. I will keep you posted when I have updates.