Exclude evicted pods from termination

[nsidhaye]

I tested your simple but effective chaoskube & I think it really useful for us.

Do you have any plans to exclude evicted pods from terminations ?

e.g. kubectl get pods -n {your_namespace} -w

cassandra-db-fc85698c7-hsw6w                      1/1       Running            0          7m
cassandra-db-fc85698c7-l57ms                      0/1       Evicted            0          4h
cassandra-db-fc85698c7-pg8ps                      0/1       Evicted            0          4h
cassandra-db-fc85698c7-swpk9                      0/1       Evicted            0          4h
cassandra-db-fc85698c7-tv4nv                      0/1       Evicted            0          4h
cassandra-db-fc85698c7-zjvwz                      0/1       Evicted            0          4h

In this case chaoskube try to evict pods which already evicted.

If this feature already implemented then can anybody guide me in configuration?

Observation: pod which are in error state or not fully started victim of chaoskube.

[linki]

@nsidhaye Thanks, I'm glad you find it useful.

I did not consider this and similar cases at all. Evicted, Terminating, ContainerCreating, CrashLoopBackoff, etc. should probably all be ignored.

Do you think it makes sense to only consider Running pods as targets? Is there a valid reason to include any other state?

[nsidhaye]

@linki : Sorry for late reply. I feel there is no point to killing pod which is already evicted or terminating & even CrashLoopBackoff state.

If we target only Running pods then we can effectively use chaoskube in chaose engineering. Assume chaoskube run interval is set to 120 min. If chaoskube is trying to kill pods which are not in healthy state then we are loosing one opportunity to create chaos.

Even better if we can configure state of pods. In case configuration then we are not changing default behavior of chaoskube.

[Aergonus]

kubemonkey has an option to kill-all types vs just running pods. It can be trivial to add but I don't think there's a use case for terminating evicted pods. I would say you should be using some kind of pod-reaper to remove evicted pods.

[nsidhaye]

@Spellchaser : I don't want to spend cycles on trying to delete evicted pods. Instead I want chaoskube should only consider healthy pods to terminate.