Expand docs and hardware support for recent Intel NUCs

zenmonkeykstop commented 4 years ago

Description

7-series NUCs were discontinued in mid-April and are no longer officially available from Intel (tho probably still sporadically available at retail for a while). The recommended hardware list should be updated to include NUC models that can be reasonably expected to be available long-term. Given that 10-series NUCs are now available, it may make sense to add both 8-series and 10-series NUCs to recommendations.

The last update had issues with Ethernet chipset support in the kernel. The 8- and 10-series NUCs use the same chipset (Intel i219-v) as the 7-series, so this will probably work, but hardware testing is still required to be safe..

Tasks

[x] acquire NUC8i5BEH and NUC10i5FNH kits, RAM, storage
For each:
- [x] verify Xenial server installation works, system boots with default kernel without error
- [x] install latest SD grsec kernel, verify system boots with new kernel without error
- verify kernel builds
- [x] verify that changes not required for NUC8
- [x] flag 5.4+patch requirement for NUC10
- [x] update documentation references
- [x] update QA matrix to include NUC8
- [ ] Provide a 5.4 kernel with NUC10 support - #5479
- [ ] Add NUC10 to hardware recommendations
- [ ] Ensure that BIOS update instructions are applicable to NUC10: https://docs.securedrop.org/en/stable/update_bios.html
- [ ] Add NUC10 to QA matrix template

User Research Evidence

(Feedback from folks looking into setting up instances)

eloquence commented 4 years ago

Still awaiting hardware - deferring for current sprint/1.5.0, we'll see where we are mid-sprint.

eloquence commented 4 years ago

Hardware obtained. For the 8/20-9/2 sprint, @zenmonkeykstop committed to step through and document the install procedure on Ubuntu 16.04, in the second half of the sprint.

zenmonkeykstop commented 4 years ago

Initial testing done against NUC10i5FNH, Ubuntu 16.04.6 server install:

onboard NIC not detected (as expected) so a generic USB Ethernet adapter was used instead
after installation and update to latest Ubuntu 4.4 series kernel, NIC still not detected.
downloaded and installed latest SD grsec kernel (4.14.188) - NIC still not detected.
did a bit of research, looks like despite the fact that the same chipset is listed as for NUC7s, there is an updated serial code somewhere that is not included in the Kaby Lake patch we're applying at kernel build time. 10th-gen NUCs are supported under Linux 5.5, and it looks like support was also backported into the 20.04 Focal kernel.

eloquence commented 4 years ago

Discussed in sprint planning today. We've got NUC8s on order and will test those in late September. The NUC7s are still available in retail (including online: Amazon, Newegg, etc.) but getting stale.

rocodes commented 4 years ago

I have a NUC8 and have (briefly) gone over the initial testing with @zenmonkeykstop. I'll plan to step through and document results as well during the coming sprint.

eloquence commented 4 years ago

Per above @rocodes will do some testing on the NUC8 during the sprint, but we're not aiming to QA 1.6.0 on it yet.

rocodes commented 4 years ago

Partial update on NUC8 hardware testing:

[x] Ubuntu 16.04 LTS installed successfully. NIC was not recognized during installation wizard, nor was the USB ethernet adapter (Tripp-Lite USB 3.0 Adapter U336-000-GBW, and I also tried an Asus adapter I had lying around). However, once Ubuntu was installed, a quick lspci indicates that the ethernet adapter device is actually detected, but the interface wasn't configured, so this is quickly fixable and the USB ethernet adapter is usable in Ubuntu. (I didn't spend any time trying to figure out if it was possible to get the NIC recognized in vanilla Ubuntu since I wanted to focus on grsec, but I could do).
[ ] securedrop-grsec-4.14.188 kernel installation not completed. Installing the linux-headers-4.14.188-securedrop-grsec and linux-image-4.14.188-securedrop-grsec occurs without issue, but the securedrop metapackage requires a fallback kernel, and in the apt repo, we only seem to have linux-image-4.14.175 but not the headers package, so the installation of 4.14.175 is not successful. Therefore, the securedrop-grsec-4.14.188 metapackage cannot successfully be installed (it complains about a missing dependency).
[ ] Booting into the grsec 4.14.188 kernel from the GRUB menu, it currently hangs. This sounds like an issue similar to what @zenmonkeykstop encountered historically with the NUC7s, which required a kernel update to mitigate. Conducting further diagnostics to gather more information.
[x] Ubuntu 20.04 Focal installs and boots successfully and ethernet connection is detected successfully without the use of a USB adapter.
[ ] Compiling and installing a grsec-5.4.68 kernel (back on Xenial on the NUC8) is also unsuccessful, and hangs at initrd in the same way the 4.14.188 kernel did. (Thanks to @conorsch for much assistance with this and other steps!) Also under conor's watchful eye, the problem persists after pulling in the two Kaby Lake patches from https://github.com/freedomofpress/ansible-role-grsecurity-build/tree/main/files and recompiling this kernel. I can confirm that the kernel boots on a Xenial-based qube, so it appears to have been compiled correctly.

rocodes commented 4 years ago

Unfortunately, even after building a new 5.4.68 kernel based on a new kernel config provided by @emkll today, I'm unable to boot into it on the NUC8, though I can boot into it on a Xenial qube, same as with the kernel I built yesterday.

I do notice the following console output when installing the kernel: [...] Ignoring non-Xen Kernel on Xen domU host: vmlinuz-5.4.68-grsec

Some quick searching tells me this is grub trying to be clever, however, a) the kernel still shows up in the grub menu and b) this message is also displayed in the Xenial qube when installing the kernel, which then goes on to boot just fine.

Stay tuned folks... Sorry it's not a happier update.

eloquence commented 4 years ago

@rocodes and @zenmonkeykstop will continue their work on this in the 10/1-10/15 sprint, aiming for no more than 16 person hours total time spent during this sprint. Our goal right now is to get the NUC8s running on a 4.14 series grsec kernel, so we can potentially ship support in a point release or 1.7.0. Additional 5.4 testing is out scope for now, but we'll revisit 5.4+ kernel support as part of the Focal migration (#4768) at the latest.

rocodes commented 4 years ago

At @emkll's suggestion I tried compiling a 4.14 series kernel without grsec patches to see if it would boot on the NUC8, thereby helping us determine whether the issue is with the grsec patches or not. Unfortunately, a 4.14.188 kernel without grsec patches does boot successfully on the NUC8, while the 4.14.188 kernel from the apt repo that includes grsecurity patches does not.

But wait there's more! In both cases (the kernel that successfully boots and the one that does not), during installation I see the following warning:

W: Possible missing firmware /lib/firmware/i915/kbl_guc_ver9_14.bin for module i915
W: Possible missing firmware /lib/firmware/i915/bxt_guc_ver8_7.bin for module i915

However, installing the appropriate drivers has not enabled the grsec kernel to boot.

zenmonkeykstop commented 4 years ago

tested on the NUC8i5BEK as provided by @rocodes:

16.04.7 installer with built-in NIC: NIC not detected
16.04.7 installer with USB Ethernet dongle (TP-Link UE300): USB NIC detected, installation as normal
installed 4.14.188-grsec kernel+headers from apt-test with dpkg and rebooted: boot frozen after loading initial ramdisk
added debug ignore_loglevel earlyprintk=efi,keep to grsec kernel parameters for more info - found boot freezes after ACPI: 8 ACPI AML tables successfully acquired and loaded
futzed around for a bit with no joy, but then...
...checked kernel parameters for a working grsec install.
added missing mds=full,nosmt noefiflags: boot successful!
verified built-in NIC support

tried out an SD install with the dongle:

install proceeded as normal with default settings
after ./securedrop-admin install sshed to app server and switched default interfaces in /etc/network/interfaces
rebooted but had a bit of a chicken-egg moment, given potential iptables woes decided best procedure would probably be:
- install with ssh enabled
- switch NICs as above
- rerun sdconfig to switch to ssh-over-tor and rerun install playbook.

With a working USB adaptor, it looks like the NUC8s are a viable install option, though it would be worth going through a full install run as above to verify and document the process.

(One thing to note, paxtest tests pass but one meltdown test (for the Foreshadow SGX vulnerability mitigation) fails.)