search

karafka / waterdrop

Standalone Karafka library for producing Kafka messages
https://karafka.io
Other
257 stars 39 forks source link

Trying to understand recommendation against at_exit #455

Closed jlambert121 closed 8 months ago

jlambert121 commented 8 months ago

The FAQ states that at_exit is not recommended for shutting down WaterDrop, but does not give any context as to why. Would you be able to provide more context as to why this recommendation?

The specific use case we are running into is specifically with sidekiq. We are unable to send a USR1 signal to just quiet the queue because of limitations within our environment, just a SIGTERM. Adding Karafka.producer.close to the :shutdown event runs it immediately when the SIGTERM is received rather than as sidekiq is exiting meaning some jobs finishing up no longer have a connection from the producer to write to. My solution was going to be to wrap the shutdown with at_exit, but wanted to understand any concerns with this solution.

mensfeld commented 8 months ago

It is not recommended because it is not fully predictable

old but still valid: https://blog.arkency.com/2013/06/are-we-abusing-at-exit/

I recommend using given framework native exit capabilities. It no native capabilities are present and you know what you are doing at exit should be sufficient.

Older versions of karafka rdkafka would occasionally crash on at exit close but I did not see this issue for years now so you should be good.

jlambert121 commented 8 months ago

Perfect - thank you for the response and clarification!