Restore FDE Support with Automatic kexec Reboot

[eviljoel]

I figured out a way to reboot a full disk encryption (FDE) system without requiring an administrator to reenter a password. This method should work even with automatic system updates. It works by extracting the disk encryption key from ram, storing it in a custom initrd on a ramdisk, and booting directly into the latest Linux kernel with kexec.

Based on issue https://github.com/freedomofpress/securedrop/issues/511, the main reason you are not recommending FDE is because automatic upgrades become infeasible. Typically, a system administrator must retype the disk password to unlock a system after an automatic update. This typically results in significant downtime or non-updated systems. I believe the code in this issue largely addresses this concern.

I've created a very rough script to demo this functionality. To run it, first apt-get install the "kexec-tools" package. You do not have to enable the reboot support when it prompts you. Next run the following script on a default Ubuntu 14.04 install with full disk encryption as root:

#!/bin/sh

version=`grep "menuentry 'Ubuntu, with Linux " /boot/grub/grub.cfg | cut -d " " -f5 | cut -d "'" -f1 | head -1 | tr -d '\n'`

mkdir /tmp/tmpfs
mount -t tmpfs none /tmp/tmpfs
cd /tmp/tmpfs
mkdir initrdtmp
cd initrdtmp
dmsetup --showkeys table | awk '/^????_crypt: /{ print $6 }' | tr -d '\n' | perl -e 'print pack("H*", <STDIN>);' > key
cat /boot/initrd.img-$version | gunzip | cpio -id
sed -ri 's/\$cryptkeyscript \"\$cryptkey\" \| \$cryptcreate --key-file=- ; then/\$cryptcreate --master-key-file=\/key ; then/' scripts/local-top/cryptroot
find . | cpio --create --format='newc' > ../initrd
cd ..
gzip initrd
kexec -l /boot/vmlinuz-$version --initrd=initrd.gz
shred -zu initrdtmp/key
shred -zu initrd.gz
cd /tmp
umount /tmp/tmpfs
rmdir tmpfs
kexec -e

Unfortunately, this probably won't work on all systems. Some drivers expect hardware to be in a post BIOS, uninitialized state and will not work without a reboot. Therefore this type of system reboot should remain optional.

Credit: I grabbed a line of code from here: https://www.debian-administration.org/users/dkg/weblog/71

[garrettr]

@eviljoel First, apologies for the delay in responding to this. I took the last week off for holidays with my family.

This is a very clever solution! I have not had a chance to test it, but believe it works. My primary concerns with this approach are:

1. Does this work with grsecurity? Starting with the upcoming SecureDrop 0.3 release, all SecureDrops will run grsecurity kernels by default. The security benefits of a hardened kernel are tremendous, and we would not want to trade that off for automatic rebooting with FDE, especially given what we see as the limited utility of FDE for SecureDrop (it's a server that's always on, all serious adversaries are able to use that to circumvent the protections offered by FDE).
2. As you say, this "probably won't work on all systems". It would definitely need to be optional, since we have enough problems with hardware compatibility (mostly due to grsecurity) already.

[eviljoel]

@garrettr Hmm... I researched this and I'm almost certain it will work if you disable "Deny reading/writing to /dev/kmem, /dev/mem, and /dev/port" and enable "kexec system call". The "Deny reading/writing to /dev/kmem, /dev/mem, and /dev/port" option can be found under "Security options" -> "Grsecurity" -> "Grsecurity" -> "Customize Configuration" -> "Memory Protections". The "kexec system call" can be (conditionally) found under "Processor type and features".

It kinda makes sense that some memory protections have to be disabled for kexec to work. I'm somewhat of the opinion that it is an acceptable trade off to disable these protections. Our adversaries might mess up when taking possession of SecureDrop systems. Also, adversaries in other countries might not be as well trained. I'm not sure the benefit of FDE is trivial.

I guess you'll have to evaluate the costs vs the benefits. What is the value of this memory protection vs the benefit of unattended FDE reboots?

Anyway, even with the modified Grsecure kernel, I haven't quite gotten the script to work yet. However, I have gotten a modified Grsecure kernel kexec reboot to work so I think I'm only a few hours away from debugging the above script.

[garrettr]

I'm somewhat of the opinion that it is an acceptable trade off to disable these protections.

Yeah, this could be difficult.

A potential problem that I can anticipate is supporting the alternative grsecurity kernel (that disables some of the hardening options in order to make grsecurity work) in our package repository. All SecureDrops starting with 0.3 will use the Freedom of the Press package repository by default, and we will be responsible for providing upgraded versions of the grsecurity kernel on a timely basis. Since building this kernel in a safe/paranoid way is a huge pain in the butt, and one that needs to be incurred fairly often (grsecurity typically gets updated every few weeks), we may not be willing to additionally support an alternate version of the kernel for this use case.

You could certainly operate your own package repository for the FPF packages, and be responsible for maintaining your customized grsecurity kernel, but it would be up to you to keep everything up to date with both your custom grsecurity and the rest of the SecureDrop packages (for the web application, ossec monitoring software, etc.)

However, if we can integrate this in a reasonable way that doesn't incur too much management overhead for us, I'm all for supporting it.

[eviljoel]

So I figured out what was wrong with my script. Our test builds are still running on Ubuntu 12.04. Ubuntu 12.04 ships with cryptsetup 1.4.1. We need cryptsetup 1.4.2 to get the --master-key-file option working with luksOpen. I'm confident that this is the problem and am not going to install another test build just to see the above script work (unless you really want me to).

While I would certainly like you to maintain two builds, I can understand not having the time to support two kernel configurations. Maybe you can automate your build process to support building both versions at the same time? Are your automated build scripts open sourced? If so, I might try to hack something in there to support this.

As for maintaining our own repo, we are considering all our options. Thanks for the suggestion there.

[garrettr]

@eviljoel Ok, this all sounds good. I think we should revisit this after the 0.3 release is done, and consider it as a feature for 0.4. I am setting up the Github metadata accordingly, let me know what you think.

[heartsucker]

Marking this "Pending close" as this is way too complicated to guarantee to work on all systems, and it's likely not in the scope of the threat model. This is more likely to cause support problems than solve problems.

[conorsch]

This is more likely to cause support problems than solve problems.

Bingo. Definitely interested in supporting FDE one day, but it's not exactly common on bare metal servers. Would be more interested in confidently destroying memory so that we don't need to fall back to forcing a nightly reboot to minimize forensic discovery about plain text machines on seized servers.

[eaon]

Reopening this as we're currently exploring FDE options. (Ironically, this is an idea that came to me tonight and I found this issue by researching grsec and kexec compatibility :laughing:)

As far as I can tell from the reading that I've done, the memory protection issues have been resolved upstream and and kexec might work with the kernel configuration we currently have. If that's the case, this could be a viable path for TPM-less FDE support. It wouldn't support recovery from loss of power, but for unattended upgrades (especially with the limited recommendations for hardware that we have these days where we can check for driver issues), this is worth revisiting.

PS: I'd probably opt for adding a keyfile slot during provisioning instead of grabbing the "master key" from memory.

[eaon]

Because we have CONFIG_KEXEC_FILE enabled already, I can confirm that we are able to "soft reboot" with kexec if we use the -s argument, instructing it to use the file based syscall rather than the memory one.

I successfully tested this with a grsec kernel we're about to release and its respective initramfs. kexec -e worked as expected, while systemctl kexec ended up crashing, I assume because the files I instructed kexec to load were not available anymore after all the unmount operations. So there's still some investigation left to do to make it work well but I'm happy to report that this is a viable option.

[ghost]

Some update on this: I was able to get SD to successfully reboot with FDE using a kexec approach. However, this was in a VM environment, and I would need to verify that it works properly on actual hardware. There's still more work to do around verifying that the approach is stable and usable in real-world setups, but it's a promising start

[ghost]

Quick update: I've since reproduced the results on a hardware setup