search

FoundationDB / fdb-kubernetes-operator

A kubernetes operator for FoundationDB
Apache License 2.0
241 stars 82 forks source link

Operator should recreate Pods if they are not fully excluded #970

Closed johscheuer closed 2 years ago

johscheuer commented 2 years ago

Currently when a process group is marked for removal and someone deletes the Pod the Pod won't be recreated. Instead of only checking if the Pod is removed we should check if the Pod is marked for removal and is fully excluded (or exclusion is skipped).

Code part: https://github.com/FoundationDB/fdb-kubernetes-operator/blob/master/controllers/add_pods.go#L68

That could help in cases where someone bring the cluster in a bad state and marks to many Pods with removal.

brownleej commented 2 years ago

This could overlap with #309

brownleej commented 2 years ago

We should also consider whether there are cases where this behavior isn't desireable, like if recreating a pod could cause problems for the cluster. In scenarios like that, it would be good for users to be able to shut down a pod without it being fully excluded and rely on the operator not recreating the pods.

johscheuer commented 2 years ago

I see there's an overlap with 309 but there are some other use cases 309 could solve that are not the target here. If a user wants to delete a Pod and ensure that it won't be recreated they just have to put it on the WithoutExclusion and then delete it. Since the user doesn't care about the exclusion that actually expresses the desired behaviour very well.