Set up continuous integration for dom0 tests

[redshiftzero]

Description

The tests we have are executed from dom0 in Qubes, so we should consider how to rig something up in CI. We should check out what Qubes does for CI across their repos and see if we can adopt a similar approach: https://github.com/QubesOS/qubes-core-admin/blob/master/.travis.yml https://github.com/QubesOS/qubes-core-admin/blob/master/run-tests

Relevant upstream Qubes issue: https://github.com/QubesOS/qubes-issues/issues/482

Requirements

• We want to block PR merges into main for the securedrop-workstation repo until make test is run and passes in CI (if make test turns out to be too flakey, we'll consider not blocking but that is a decision we can delay for now)
• We want to install the securedrop-workstation package in a clean environment and run CI any time a new securedrop-workstation package lands on apt or apt-test and report back (perhaps in a notifications channel)
• We want make test to run in a clean environment and report back the result to github automatically

[redshiftzero]

I talked with Marek today at FOSDEM about how this is done over at Qubes and he described that they have:

1. Travis CI running unit tests and qubes-builder tests (that's what you see in the links in the original issue ^)
2. All integration testing (that's what we'd want for this repo) done via manually kicking off a test run on physical hardware.
3. Qubes install testing done via OpenQA - OpenQA interacts with the installer (clicks buttons, etc.), takes screenshots at each stage of the install, and then uses fuzzy image matching to compare the observed screenshots with the expectation. For Qubes, this is used by running the installer using KVM. Check this thread out for more details.

[eloquence]

2022-08-10 review: Issue title updated for clarity (we have limited CI for other tests, but not for the dom0 tests).

[sssoleileraaa]

Here's a quick update after day one of looking into this issue:

• Maeve, Erik, and I met to strategize, and we also had our first round of monthly meetings with Marek and Marta today

• Our first goal is to automate our dom0 unit tests (e.g. checks that verify expected RPC policies) against a freshly-provisioned workstation on Qubes OS

  • Maeve is looking into how we automated SecureDrop server tests that use nested virtualization on Google Cloud

  • Allie is looking into how to build a Qubes OS image with nested virtualization enabled

  • The Qubes team uses Open QA, which is perfectly designed for their needs as they are testing an entire operating system, including the installer. The tests take over 24 hours to run. The team does not use a Qubes OS image - it sounds like they run the installer every time

  • We asked Marek about whether or not crashing is still an issue when HVM is the virtualization type and he said it should work fine (our concern stemmed from the docs which mentioned crashing: https://www.qubes-os.org/doc/automated-tests/#automated-tests-with-openqa)

  • We're still in the research and discovery phase so not too much to report yet

[sssoleileraaa]

We're making progress towards our goal to automate dom0 unit tests (we deviated a bit from our original plan and did a lot of side-by-side troubleshooting, repro'ing of issues, and comparing results). Here's an update before our extended holiday break.

What we accomplished:

• :heavy_check_mark: We were able to figure out how to enable nested virtualization on Google Cloud as well as several different virtualization software, including VMWare, VirtualBox, and qemu
• :heavy_check_mark: We were able to create a local Qubes image with nested virtualization enabled, so that we don't need to run the Qubes installer every time we run CI

What's left towards our first milestone?

• At this point we are focused on migrating our Qubes image to the cloud. Research time is needed to read more about https://cloudinit.readthedocs.io/en/latest/index.html and to investigate why exactly the tooling needs network access because we'll need it to run in dom0 when the VM starts, but dom0 doesn't have network access.
• After some time-boxed research into cloud-init, we'll want to decide on whether to continue down the cloud solution path or just use bare metal servers as the Qubes team does for their CI infrastructure.
• Worth noting: We investigated using qemu and VirtualBox to create a base Qubes image for migrating to the cloud, but we saw the most success with using VMWare. We now have several different .vmdk files at this point which can also be imported into qemu but then you'll run into an IOMMU support issue (adding an extra qemu flag -device intel-iommu,intremap=on did not work for us and we decide to table making Qubes conditionally allow PCI passthrough on PV without IOMMU support- a less secure option but it could be informative in our troubleshooting to try this later if we have more time). There are a couple different issues with VMWare where the Qubes image eventually stops working during the securedrop-workstation installation process. Sometimes the window goes dark and there's no way to interact with it, and sometimes VMWare just starts crashing each time you try to start an image. If the cloud solution works then we might not need a workaround for this.

[sssoleileraaa]

Current state is that https://github.com/freedomofpress/securedrop-workstation-ci/issues/2 and https://github.com/freedomofpress/securedrop-workstation-ci/issues/3 need to be resolved before we can hand this over to the SecureDrop team and close this issue.

[sssoleileraaa]

Per Infra Sync discussion, this is in a waiting status due to waiting on hardware location migration

[sssoleileraaa]

The hardware handoff is happening end of month

[sssoleileraaa]

This is done! Any improvements such as adding redundancy can be added as issues to the securedrop-workstation-ci repo.