Option for disabling core dumps

[edomora97]

It may be useful to add an option for disabling the core dumps for the sandboxed process. One could say that it is sufficient to use RLIMIT_CORE, but this is not the case if /proc/sys/kernel/core_pattern is using a pipe. In fact, if /proc/sys/kernel/core_pattern starts with a pipe (|), the rlimit is ignored.

According to the Archwiki a more reliable way is by using prctl.

Note that the documentation regarding this is a bit confusing.

---

Why is this useful?

We spent the last few days debugging a strange problem with a sandboxed process under isolate. The program crashed (with SIGSEGV), but isolate was not able to kill it. Actually, we weren't able to kill it either (kill -9 pid wasn't working). The process was not a zombie, and according to htop it was "running", 100% of a core and 100TB+ of virtual memory. We are still investigating why it reports that huge virtual memory usage (probably it was the cause of the segfault). Also note how the CPU time is still climbing even after various SIGKILLs sent to it.

After many experiments we concluded that the cause is core dumping. Disabling the core dumps system-wise makes it work cleanly again. However, if isolate had a flag for disabling the core dumps just for the sandboxed process we can keep the system option to the default value.

For the record, the default option is:

kernel.core_pattern=|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h

The one we are using right now:

kernel.core_pattern=/dev/null

[gollux]

(I better don't comment on how smart is to gather all core dumps system-wide...)

Anyway, I am not sure if it ever makes sense to dump core inside Isolate's sandbox. Instead of adding an option, I suggest disabling it unconditionally.

[gollux]

However, invoking prctl(PR_SET_DUMPABLE, 0) is not going to help, because the dumpable flag is automatically reset upon exec.

[gollux]

Isolate now sets the core dump rlimit to zero. I also added a warning regarding systemd-coredump to the manual.