Requirement for kernel.core_pattern sysctl is unusual

[ADKaster]

In Fuzzer.swift on linux, there's a requirement that /proc/sys/kernel/core_pattern starts with |/bin/false, which disables all core dumps on the target system.

https://github.com/googleprojectzero/fuzzilli/blob/172778a3397774a337edee803fb372a48b99f5bd/Sources/Fuzzilli/Fuzzer.swift#L702-L714

This is a bit awkward to force on anything other than a short-lived VM of some sort. Even when running in docker, or podman or another container runtime, this kernel config value is set for the host and all containers equally, and so requires a privileged container to even set. It also overwrites the value for the host, which is awkward if it is a desktop distro like Ubuntu that expects to route crash dumps through a tool like apport.

Would it make more sense to patch the JS runtimes to set something like prtcl(PR_SET_DUMPABLE, 0) when running through fuzzili if this is the desired effect?

 PR_SET_DUMPABLE (since Linux 2.3.20)
              Set the state of the "dumpable" attribute, which
              determines whether core dumps are produced for the calling
              process upon delivery of a signal whose default behavior
              is to produce a core dump.

[carl-smith]

Yeah, I agree, we should do it in a different way. I would propose https://man7.org/linux/man-pages/man2/setrlimit.2.html. Specifically we should set RLIMIT_CORE to 0 in Fuzzilli and then let the child process inherit this limit.

RLIMIT_CORE
              This is the maximum size of a core file (see [core(5)](https://man7.org/linux/man-pages/man5/core.5.html)) in
              bytes that the process may dump.  When 0 no core dump
              files are created.  When nonzero, larger dumps are
              truncated to this size.

The benefit of using this is that we can then also use prlimit() to query the fuzzed JS engine's RLIMIT_CORE value to check whether the process is actually not dumping, and notify the user before we create tons of core dumps during fuzzing. This way we don't actually need to change any of the patches.

[carl-smith]

Fixed now with d386a83812496b7e590ce93c8fdc50208dacbd26.