Automate the creation of grsecurity template using Qubes Builder

[eloquence]

Through the work in #104 we learned that it is feasible to run grsecurity-enabled VMs as template HVMs in Qubes OS. The memory overhead of HVMs (as opposed to PVH-based VMs) is a bit higher but not prohibitively so. The next step is to automate the creation of grsecurity template HVMs using Qubes Builder. This will result in RPM packages that can be installed in dom0.

Since we don't fully understand what's involved yet, we have timeboxed this task to 16 hours for the 7/11-7/25 sprint. If it's not completed by the end of the timebox, @emkll (who will likely continue to work on this) will post an update here.

Note that this still gets us only part of the way to transitioning to grsecurity VMs (esp. for the sd-svs and sd-journalist VMs); we will still have to figure out the distribution/update strategy we want to use for these packages.

[emkll]

I've made some good progress with the template creation thanks to @conorsch 's help on the infra side.

Here is a rough walkthrough of how to build a grsec kernel debian based template vm

• Create a Fedora 26-based AppVM and expand it's storage to 20gb. Let's call it or repurpose work.

  qvm-volume extend work:root 20g
  # clone qubes builder
  git clone https://github.com/qubesos/qubes-builder
  cd qubes-builder
  #copy the template-securedrop-workstation folder in qubes-builder/qubes-src
  cd qubes-builder
  # Download sources (once we get a repo for this work, it will be pulled in through this):
  make get-sources
  #copy the example builder template in the repo root
  cp qubes-src/template-securedrop-workstation/builder.conf.example builder.conf
  make install-deps
  #somehow, not all deps are installed
  sudo dnf install perl-Digest-MD5 perl-Digest-SHA
  make qubes-vm
  make template

The template rpm package is in qubes-builder/qubes-src/linux-template-builder/rpm/noarch/

# copy to dom0
qvm-run --pass-io work 'cat /path/to/rpm.rpm` > rpm.rpm
sudo dnf install rpm.rpm

It takes 3-5 mins to install, after which you will have a securedrop-workstation template, from which you can spawn grsec patched appvms by running the following command in dom0:

qvm-create --template securedrop-workstation securedrop-workstation-test --class AppVM --property virt_mode=hvm --property kernel=''

Currently the template itself does not have hvm set as virt mode, and kernel is set to the qubes default. I can't seem to find a workaround at this time.

[conorsch]

Struggling to reproduce the results here. A bit of clarification on the setup instructions is warranted. Perhaps we can pair on this briefly, @emkll, to tag-team the documentation in this issue? If you and I both agree the docs are sufficiently clear, then we can turn it over to the rest of the team to reproduce.

At a high level, here are some of the problems I encountered:

• There's no mention of running the ./setup script within the qubes-builder repo, but I assume that's in fact necessary, in order to fetch signing keys and other friends. Perhaps I'm mistaken in that requirement, though.
• qvm-volume extend work:root 20g fails for me. Inspecting the journal in dom0, I see a Python exception regarding int and str comparisons. Various permutations such as "20G", "20GB" (taken from the docs) did not resolve. Didn't run out of disk space, but suspect I will once the builds get rolling. Will try to reproduce this on subsequent boots. Made sure that dom0 packages are up to date, and tried with the work VM started and halted (I assume halted is a requirement!).
• There are two cd qubes-builder commands in the testing instructions. The second should likely be omitted—unless it should be cd qubes-src, but I suspect not.
• The filepath for builder.conf I found in the WIP template repo was securedrop-workstation.conf, not builder.conf (the README in the WIP repo clarified this somewhat).
• make get-sources tries to fetch https://github.com/QubesOS/qubes-template-securedrop-workstation.git, which does not exist. It appears we need to override the URL somehow, perhaps via the GIT_URL_<x> env vars.

Happy to sit down with you and piece the bits together. Perhaps after standup tomorrow, we can pair in real-time to sort it out.

[conorsch]

Progress! Paired with @emkll briefly today and sorted out a few blockers. Will outline some clarifications here for others trying to reproduce the work above.

Volume resizing woes

As reported above, I was unable to resize the root partition on in the "work" VM on my Qubes system. Even after installing all available dom0 updates (using stable repos), the recommended volume resize command showed an error. After patching locally, I got past the failing type error, but was informed that the root partition on the work VM was readonly. Details below.

Detailed errors on qvm-volume resize operation

[user@dom0 ~]$ qvm-volume resize work:root 20g Got empty response from qubesd. See journalctl in dom0 for details. [user@dom0 ~]$ qvm-volume resize work:root 21474836480 Got empty response from qubesd. See journalctl in dom0 for details. From `journalctl -a | tail -n 100`: Jul 18 14:54:06 dom0 qubesd[2213]: Traceback (most recent call last): Jul 18 14:54:06 dom0 qubesd[2213]: File "/usr/lib/python3.5/site-packages/qubes/api/__init__.py", line 262, in respond Jul 18 14:54:06 dom0 qubesd[2213]: untrusted_payload=untrusted_payload) Jul 18 14:54:06 dom0 qubesd[2213]: File "/usr/lib64/python3.5/asyncio/futures.py", line 381, in __iter__ Jul 18 14:54:06 dom0 qubesd[2213]: yield self # This tells Task to wait for completion. Jul 18 14:54:06 dom0 qubesd[2213]: File "/usr/lib64/python3.5/asyncio/tasks.py", line 310, in _wakeup Jul 18 14:54:06 dom0 qubesd[2213]: future.result() Jul 18 14:54:06 dom0 qubesd[2213]: File "/usr/lib64/python3.5/asyncio/futures.py", line 294, in result Jul 18 14:54:06 dom0 qubesd[2213]: raise self._exception Jul 18 14:54:06 dom0 qubesd[2213]: File "/usr/lib64/python3.5/asyncio/tasks.py", line 240, in _step Jul 18 14:54:06 dom0 qubesd[2213]: result = coro.send(None) Jul 18 14:54:06 dom0 qubesd[2213]: File "/usr/lib/python3.5/site-packages/qubes/api/admin.py", line 459, in vm_volume_resize Jul 18 14:54:06 dom0 qubesd[2213]: yield from self.dest.storage.resize(self.arg, size) Jul 18 14:54:06 dom0 qubesd[2213]: File "/usr/lib/python3.5/site-packages/qubes/storage/__init__.py", line 481, in resize Jul 18 14:54:06 dom0 qubesd[2213]: ret = volume.resize(size) Jul 18 14:54:06 dom0 qubesd[2213]: File "/usr/lib/python3.5/site-packages/qubes/storage/lvm.py", line 439, in resize Jul 18 14:54:06 dom0 qubesd[2213]: if size < self.size: Jul 18 14:54:06 dom0 qubesd[2213]: TypeError: unorderable types: int() < str()

The command that worked in the end was:

qvm-volume resize work:private 8589934592

That's about four times the default private volume size. Used that number when it appeared that twice the normal size wasn't sufficient (disk utilization on the rw partition inside the work VM passed 95% during build).

Builder dependencies

The make get-sources command initially failed for me when trying to fetch sources for the securedrop-workstation template. Was able to resolve (with assist from @emkll) by first copying in the example config for the Debian builder, running make get-sources to fetch all the necessary files, then copying in the securedrop-workstation config and proceeding with the instructions above. For clarity:

git clone https://github.com/qubesos/qubes-builder
cd qubes-builder
./setup # Appears this is required, worth confirming
cp example-configs/debian.conf builder.conf
make install-depends
sudo dnf install perl-Digest-MD5 perl-Digest-SHA # May be worth a PR...
make get-sources

# Now that we have the Debian sources, let's switch to the workstation template.
# The repo is temporarily private, thus the git-over-SSH command.
cd ..
git clone git@github.com:freedomofpress/qubes-template-securedrop-workstation
cd qubes-builder
mkdir -p qubes-src
cp -r ../qubes-template-securedrop-workstation qubes-src/template-securedrop-workstation
cp qubes-src/template-securedrop-workstation/securedrop-workstation.conf builder.conf
make get-sources
make qubes-vm
make template

The final make template command failed for me, but the issue is local to my workstation: I've still got to provision access to the staging apt repo used for testing. Will test again once I do the needful in that regard, and report final results in this ticket.

[conorsch]

Success! Fussed with volumes for a while until extension was successful; Qubes rewards perseverance. Was able to generate a ~700MB RPM, which expanded to ~3GB upon install in dom0. Created a new VM as instructed above, and confirmed that the latest grsec kernel was indeed running. Also confirmed network access as a sanity check, but performed no other investigation into VM functionality.

As far as the "automation" spirit of this ticket, we have the process documented in this discussion thread, but no easily reproducible script. I suggest:

• creating a quick-and-dirty bash script reproducing the steps in @emkll's instructions above (see clarifications in follow-up comment as well)
• storing said script in this repository, rather than creating yet another

Thoughts?

[emkll]

Thanks @conorsch for the careful testing, I agree we should write a script to wrap the builder commands. I will try to get a PR up against this repo before the current sprint ends.

I have created some follow up tickets as well as updated some existing tickets based on research work done in this task:

• Improvements to packaging. We will want to wrap the Linux kernel into a metapackage. This improvement in packaging will permit us to build templates more infrequently and reduce configuration drift (#113).
• We should investigate further hardening opportunities (see #21) :
• We should look into adding packages frequently used by journalists into the template (and adding the apt repos), like MAT for example (see #112)
• We have yet to make a final decision on if we are using Debian or Fedora based VMs, the discussion is being tracked in #111

[emkll]

Build logic for the template is now available in this repo: https://github.com/freedomofpress/qubes-template-securedrop-workstation, and a build script is introduced in #115 .

[kushaldas]

Ensure your .gitconfig contains qubes-gpg-client-wrapper and import the Qubes Master Key (https://www.qubes-os.org/security/verifying-signatures/) :

^^ this is in the README file of the new repo, but it does not tell where this has to be done. In dom0 or in a new appvm.

[emkll]

Good point @kushaldas , these public keys are needed in the AppVM that builds the template (or the split-gpg VM, if applicable). At the moment this script generates unsigned. Will update the docs