search

arches / whacamole

restart heroku dynos that run out of RAM instead of swapping to disk
115 stars 12 forks source link

Add a warning about preboot in README #13

Closed michaelbaudino closed 9 years ago

michaelbaudino commented 9 years ago

This PR adds a warning in README about using Whacamole on an application that has preboot enabled.

We've seen some weirdness on a 30+ dynos app and Heroku support engineer @watsonian found this out (all credit goes to him, actually):

It looks like Whacamole might not be behaving well with preboot. The problem is that there's a 3 minute period where both the old and new dynos are running. The old dyno is still processing requests while the new dyno is booting. The memory usage log emissions that Whacamole is seeing are from the old dyno, not the new dyno. Whacamole is then restarting your dyno even though it's already in the process of rebooting. This leaves your old original dyno (that's swapping) running for even longer because the new restart spins up yet another dyno to preboot (which takes ~3 minutes).

Note that the second issue I mention here might possibly be fixed by maintaining a trace of not only the dyno name (web.X) but also their unique id (dyno=heroku.28023208.ab6ae82e-64b3-41e0-84d5-dc35124b113d in the logs).

michaelbaudino commented 9 years ago

The CI errors don't seem to be caused by this PR :confused:

arches commented 9 years ago

thanks for this! dyno ID fix sounds like a good approach. not sure when I'll be able to get to it myself though.