mariadb was oomkilled

[pushcx]

In #lobsters eeeeeta suggested using OOMScoreAdjust to prevent mysql from being oomkilled again.

Probably we want to turn that score down for mysql, nginx, and/or the master unicorn process, and/or up for unicorn workers. I don't know how much to change these numbers by or where it would be most effective with fewest side effects. Anyone familiar?

[jstoja]

I personally wouldn't touch the OOMScore since the OOMKiller might want to kill system processes (like I don't know sshd or the dns client) making it very annoying to restore anything.

I would rather suggest using the builtin ressource control from systemd (that is using cgroups to do that). It should be easy enough to implement and I would consider this a better practice than adjusting the score.

What do you think about it?

[pushcx]

One of the nice things about prgmr is that we have a serial console we can connect through if ssh or other public networking breaks.

Could you explain more of what you were thinking with this resource control? If it's setting a max for RAM, how will that play out? Will mariadb see a malloc fail? Does it respond by evicting some old cache or printing an informative error message to the logs and exiting?

[jstoja]

So there's 3 main settings for memory control: low, high and max.

• Low is the guaranted memory you set (the addition of all low should obviously be less than your total ram) Usually this is set to a bit more than what the process should use. The idea is that If another process is trying to take all the memory, you know that this process will have this guaranted anyway.
• High is the somehow soft limit to which the process will start to have memory reclaimed and have trouble allocating memory. The OOM killer will never be invoked at this point. Important note is that when not set, this is set to max, so better set it.
• Max is the hard limit where the if the high limit didn't suffice, it'll actually trigger the OOM killer.

Setting up the 3 values for your main process allow you to have "guaranted" process, MariaDB would then be "protected" from the OOM because it would stay under its hard limit, and on the contrary if Unicorn is getting way too much memory, it would reach the hIgh limit that would probably make it harder to reach the max, and if it does reach max then it's probably better to restart it than to kill another process (that might be important) and still allow unicorn to expand more.

[pushcx]

This is probably fixed now, see comment in linked PR. Leaving this open a month or two to confirm.

[pushcx]

Been more than a month, zero issues. Thanks to Scout on this one.