zombocom / puma_worker_killer

Automatically restart Puma cluster workers based on max RAM available
737 stars 77 forks source link

Implications for long-running requests #18

Closed julik closed 8 years ago

julik commented 8 years ago

I would like to try puma_worker_killer in my setup but I am using Puma to server very long requests (they sometimes span hours or even a couple of days). I cannot abort a running request mid-flight, so I need the killer to wait until the worker exhausts it's current requests, and take into account that this can take long.

Is the gem usable for this sort of use case?

ruckus commented 8 years ago

Good question. I'm looking at the source and its not clear if PWK will wait until a request finishes or it restarts it mid-flight. It appears that a request is killed mid-flight.

But I'd love someone more knowledgeable to chime in. Thank you!

schneems commented 8 years ago

PWK sends TERM immediately, it's up to puma to do the right thing and clean up correctly

julik commented 8 years ago

That I do understand. The behavior after is defined by the somewhat-underdocumented worker_shutdown_timeout in Puma proper (and afaik once a TERM is received the worker won't be receiving new requests). However, if that timeout is set to a pretty large value, will PWK try a KILL too soon, or will it overreact and attempt more TERMs if the worker does not go away in a timely manner?

schneems commented 8 years ago

Everything we send will go through the puma interface. Right now we use term from Puma and it goes through it's paces. If we send the command multiple times to my knowledge puma doesn't have any kind of a counter or keep track of all of those "terms" and do something special if they hit some threshold.