Functional testing with Nix?

[lukego]

NixOS has an impressive functional testing framework. I would like to understand how directly we can apply this to Snabb Switch.

Background on the NixOS approach:

• Theory: Automating System Tests using Declarative Virtual Machines.
• Practice: Hundreds of test suites running on the upstream NixOS Hydra.
• Example: NFSv4 test case result - click on "Report of type testlog" to see the nice log.
• Docs: How to write functional tests like these for NixOS.

The basic idea is to boot a network of NixOS virtual machines and to script their interactions.

So can we use this framework for Snabb functional tests? Examples of test cases:

• packetblaster selftest. Run a simple Snabb command, as root and with access to a PCI device, and ensure that it runs for 5 seconds.
• nfv functional test. Start two QEMU VMs with vhost-user network interfaces and a Snabb process that accesses a PCI device and connects the VMs together. Cycle through a series of configurations on the Snabb process and ensure that connectivity in the guests is as expected.
• nfv packet forwarding test. Start one packetblaster load generator (with a PCI device), one nfv process (with another PCI device), and one QEMU virtual machine. Send a continuous flood of traffic packetblaster->snabb->VM and back again.

The reasons I think we should consider building such tests on NixOS is that it would free us from managing processes directly in bash scripts, it would give us access to nice features like easy-to-browse logs, and it would save us the hassle of bootstrapping software test environments ourselves.

Question is: will it work? Could we port a simple test like the packetblaster one to NixOS as a proof of concept?

The main challenge I see is that the NixOS tests are based on booting virtual machines and running the code under test inside those VMs. This will not necessarily work for Snabb and our existing tests are all based on running processes as root directly on the host. Snabb does need to run as root for a variety of reasons and running the Snabb processes inside VMs seems like a potential performance liability, particularly when the tests themselves need to start VMs and would lead to nested virtualization.

So: Can we use this framework for running non-QEMU processes directly on the host machine as root?

cc @domenkozar

[domenkozar]

TL;DR: not without refactoring driver code to support non-QEMU "backend".

Looking at https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/test-driver/Machine.pm, all of the commands are executed via qemu monitor to the guest machine. Some are very qemu specific such as start, which would start a VM. We could refactor that to introduce "local" backend and have a subset of commands available there.

For all the tests not doing performance testing, we could just port them to existing functional testing framework.

[ghost]

Just out of curiosity, this is perl right? Is it somehow NiX specific way of automation? Could this be better implemented with libvirt and its python interface?

[domenkozar]

@nnikolaev-virtualopensystems Yes. See https://github.com/NixOS/nixpkgs/issues/5241