Research more PVF worker hardening methods

[eskimor]

E.g. basic things like stack layout randomization or Kernel space layout randomization or the currently being deployed landlock and seccomp up to full virtualization. Just to name some examples, surely more hardening options exist. Compiling a list and checking what would be useful for us would make sense. Kernel space layout randomization is something to go into the wiki for operators to enable, others might be implementable by us. In any case, we should make exploits as hard as reasonably possible - an exhaustive list of options should be a good start. Then exploring those, downsides, implementation effort, feasibility, security impact, ..

https://en.wikipedia.org/wiki/Address_space_layout_randomization

[burdges]

Isn't this imposed by modern linux?

[eskimor]

Could be, I did not do any deep research. The resources I found, suggested it might not. Anyhow, this just serves as an example. Point being, I think it is worth looking into what is available in general and deploy anything that is easy/uncontroversial for us and makes attacks harder.

[sandreim]

Maybe we could also look into recommending certain Linux distros that have better security posture/features enabled by default. It would be easier than taking whatever distro anyone installs and customizing it to meet our recommendations.

[burdges]

You can warn if cat /proc/sys/kernel/randomize_va_space reports less than 2. https://stackoverflow.com/questions/11238457/disable-and-re-enable-address-space-layout-randomization-only-for-myself

There are some details here like whether the kernel itself has address space layout randomization and likely more, not really sure.

[koute]

Here's some more Linux-specific hardening options we could use, besides the currently attempted seccomp (which for now is still wholly incomplete):

• Have the worker run in its own namespaces. Either spawn the new process through clone3 with appropriate flags or unshare once it's already running. At minimum we'd want to enable a new user namespace (CLONE_NEWUSER) since that is required to allow other namespaces to be created with non-root priviledges, and then we could probably enable all of them (CLONE_NEWIPC, CLONE_NEWNET, CLONE_NEWNS, CLONE_NEWPID, CLONE_NEWUTS; some are more useful that others and some might require some auxiliary setup to keep things from breaking)
• Run the workers in their own cgroup to e.g. limit their maximum memory usage. (Pass CLONE_NEWCGROUP to clone3/unshare. AFAIK this requires CLONE_NEWUSER namespace to be usable as non-root.)
• Close all unnecessary file descriptors and leave only the required bare minimum. (close_range syscall, or probably just iterate over /proc/self/fd and close the descriptors since we want to be compatible with older kernels)
• Hide the host filesystem. (Requires CLONE_NEWNS namespace; then we can probably do a dance with mount + chdir + pivot_root + unmount2. Could be unnecessary if we're using landlock anyway, but then this would support much older Linux kernels.)
• Drop/block all potential extra permissions and capabilities:
  • enable no_new_privs bit: prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0)
  • clear all ambient capabilities: prctl(PR_CAP_AMBIENT, PR_CAP_AMBIENT_CLEAR_ALL, 0, 0, 0)
  • set all securebits: prctl(PR_SET_SECUREBITS, SECBIT_NOROOT | SECBIT_NOROOT_LOCKED | SECBIT_NO_SETUID_FIXUP | SECBIT_NO_SETUID_FIXUP_LOCKED | SECBIT_NO_CAP_AMBIENT_RAISE | SECBIT_NO_CAP_AMBIENT_RAISE_LOCKED, 0, 0, 0)
• Clear capabilities with capset
• Set extra resource usage limits with rlimit (at least RLIMIT_NOFILE, RLIMIT_LOCKS, RLIMIT_MEMLOCK, RLIMIT_MSGQUEUE, RLIMIT_NPROC should be fairly straightforward)

(Source: I've recently been working on sandboxing for our new upcoming RISC-V smart contracts VM where I'm doing all of this by default.)

[mrcnski]

Nice ticket. I've previously looked into ASLR. rustc tries to emit hardened binaries where possible: https://doc.rust-lang.org/rustc/exploit-mitigations.html

Still, could be worth running an extra check for it: https://github.com/paritytech/polkadot-sdk/issues/637

(Source: I've recently been working on sandboxing for our new upcoming RISC-V smart contracts VM where I'm doing all of this by default.)

Nice, will try to implement these measures.

[mrcnski]

Maybe we could also look into recommending certain Linux distros that have better security posture/features enabled by default. It would be easier than taking whatever distro anyone installs and customizing it to meet our recommendations.

Could be a discussion worth having. One problem is it would be a big change to impose on validators. And lots of them run on cloud providers and have limited options for distro, kernel version, security features, etc.

[koute]

Nice, will try to implement these measures.

I will be releasing the sources to my VM soon-ish (still need to finish some refactoring); I'll ping you once I do and maybe you could reuse some of my code.

Could be a discussion worth having. One problem is it would be a big change to impose on validators. And lots of them run on cloud providers and have limited options for distro, kernel version, security features, etc.

Yeah, I don't think it would be acceptable to have a hard requirement of a very new kernel and/or userland. That said, there's a balance to be struck here. I essentially see five levels of "old" here:

• Ancient versions not supported by upstream in any way (e.g. RHEL 5, Ubuntu 12.04). It's safe to say we can completely ignore those.
• Ancient versions still supported by long term extended security releases (e.g. RHEL 6, Ubuntu 14.04). These are still technically supported for paying customers, but no one normal is going to run them. My vote would be to ignore those too.
• Old versions still normally supported (e.g. RHEL 7, Ubuntu 20.04). Some people might still run them. We should support those, but not necessarily out-of-box. (e.g. if they're missing a security feature we'd like to use a validator could refuse to start on those by default, and that could be overriden by a flag at the users' risk)
• The latest LTS release (e.g. RHEL 9, Ubuntu 22.04). We should support those out-of-box with no extra flags required to start a validator.
• The latest non-LTS release (e.g. Ubuntu 23.04). We should support those and optionally take advantage of any extra security features they might provide.

Some useful links:

• Ubuntu version history
• Debian version history
• Redhat version history

[koute]

I've also found a huge list of other potential system-wide mitigations: https://madaidans-insecurities.github.io/guides/linux-hardening.html

Not everything there is necessarily a good idea (in fact, some of the stuff there we don't want), but we could use some of that info to write our own guide.

[mrcnski]

@koute that looks really good! BTW, we do seem to have our own guide already here, though I'm not sure how many users follow it.

Also, we should perhaps be careful not to burden validators too much with requirements. I mean, the rabbithole of security is never-ending, and we should decide what is the cutoff for acceptable security and not impose more on validators beyond that. Maybe providing a pre-configured image is an option?

[koute]

Maybe providing a pre-configured image is an option?

Hm, well, how would that work in practice? People usually just rent a VM with an already installed system and don't install it themselves.

Maybe having an e.g. a sanctioned supported Linux distribution (e.g. Debian or Ubuntu) and providing a setup script for that distro which would automatically harden it would be the way to go here? Because doing all of this stuff manually is not going to be doable for the majority of people, but picking a specific distro and running a single script would.

Of course this should not be a hard requirement; in the end it's the end user's responsibility over what they're running, and we shouldn't unduly burden those people who don't want to (or can't) harden their systems. But we could e.g. detect which exact mitigations are enabled and which aren't and print out a warn!ing or something like that.