search

gajus / lightship

Abstracts readiness, liveness and startup checks and graceful shutdown of Node.js services running in Kubernetes.
Other
518 stars 32 forks source link

How to correctly handle errors?? #22

Closed ispirals closed 4 years ago

ispirals commented 4 years ago

Hi guys,

i'm a little bit confused about how we should handle a critical error, that the service can't recover from it.

Until now, I was using the signalNotReady when i catch a critical exception, thinking that kubernetes will kill my pod sending the SIGTERM and doing the graceful shutdown. But seems it doesn't work like this, because the container is not ready but kubernetes it's not killing it.

I have changed the lightship.signalNotReady() to lightship.shutdown() and now the container get killed by it self, but looking for the lightship code I see that when we use the shutdown() it's sets the signalReady to true again, so this will make k8 to route requests to this pod that is shutting down.. i'm right?

I'm confused about how I should to handle a critical error for make the pod restart graceful and ensure that not more requests are routed to this container. Any advice?

Thanks a lot!

gajus commented 4 years ago

Have you read an article that is linked in the code comments?

https://freecontent.manning.com/handling-client-requests-properly-with-kubernetes/

gajus commented 4 years ago

My question here is: If you run shutdown and your service is already not ready, why does it not shutdown immediately?

ispirals commented 4 years ago

My question here is: If you run shutdown and your service is already not ready, why does it not shutdown immediately?

The problem im facing it's my service is ready and its working so the signalReady has been sent. But a critical error appear, as example, my connection with the database has been broken.

Instead try to reconnect i just want to shutdown the service and kubernetes raise a new pod.

But... if when i do a shutdown with lightship it set serviceReady to true, kuberntes will route traffic to this pod, i'm right?

Although now that I think about it... if i dont have a database connection maybe the graceful shutdown doesn't make sense.

what do you think?

thanks!

gajus commented 4 years ago

Although now that I think about it... if i dont have a database connection maybe the graceful shutdown doesn't make sense.

Graceful shutdown does not make sense if you've already crashed.