hibiken / asynq

Simple, reliable, and efficient distributed task queue in Go
MIT License
10k stars 714 forks source link

[QUESTION] How to handle shutdowns/restarts with systemctl #911

Closed arjendevos closed 2 months ago

arjendevos commented 2 months ago

In production we compile to a single go binary and run it through systemctl. However restarts/stop don't work as they hang forever. I know we have to use kill -TERM etc but we don't know the in our CI?

Is there any docs or examples for an use-case like this?

EDIT: I now have this:

# Graceful stop
ExecStop=/bin/kill -TSTP $MAINPID
ExecStop=/bin/sleep 10
ExecStop=/bin/kill -TERM $MAINPID

I get these logs:

INFO: Stopping processor
INFO: Processor stopped
INFO: Starting graceful shutdown
INFO: Waiting for all workers to finish...
INFO: All workers have finished
INFO: Exiting

However it seems to hang on Exiting and it never completes?

arjendevos commented 2 months ago

Oke found the issue, I was running it in a goroutine. Makes sense now I think about it. Changing it to the main thread fixed everything.