search

razee-io / Razee

Overview and docs
Apache License 2.0
423 stars 36 forks source link

Liveness probe is not working #135

Closed alewitt2 closed 2 years ago

alewitt2 commented 3 years ago

our razeedeploy resources are getting into a bad state, and basically stalling out. What we expect to happen, is the liveness custom script will fail, and kubernetes will see that and restart the pod. we have seen the script works as expected, but kube isnt restarting the pod.

We either need to investigate if our script is not returning the correct error response, or look into setting up a new liveness check that kube will respect.

example liveness probe definition https://github.com/razee-io/RemoteResource/blob/master/kubernetes/RemoteResource/resource.yaml#L55-L58

example liveness script that should force kube to restart us on script failing https://github.com/razee-io/RemoteResource/blob/master/sh/liveness.sh#L18-L31

alewitt2 commented 3 years ago
  1. setup minikube or kubernetes on docker desktop setup.
  2. build docker image
    1. runs a shell script that touches file once
    2. sleeps 2hr
  3. put that docker image into a deployment yaml, with a liveness script defined
  4. run that deployment on your local test cluster and see if it works as we think it should