support PXE booting in vmtest

[vmagro]

• Our qemu setup only supports booting from disk
• Add support for a DHCP+TFTP server in antlir/vm (ideally fbtftp & dhcplb)
• Write integration tests
• This may also require being able to have vmtest tests survive reboots of the underlying VM

[vmagro]

There are two routes I see for how to implement this:

1. use something completely off-the-shelf like dnsmasq
2. write small shims to use mostly-Facebook infrastructure directly in antlir/vm contexts

Using dnsmasq has its own set of challenges, but can be a useful starting point. You can start with just using a dnsmasq installation from the host (see vmtest sidecar support https://github.com/facebookincubator/antlir/blob/962d50367b42c49e1c30051b6f4451b89e99dede/antlir/vm/tests/BUCK#L194) AFAICT dnsmasq supports dhcpv6 address ranges / router-advertisements for SLAAC as well as setting DHCPv6 options including bootfile urls. It can also serve tftp directly, so it's really an all-in-one solution

Implementing from scratch might be more interesting, but requires more upfront investment before seeing results. I have left some notes about how that could be implemented below:

https://github.com/facebookincubator/dhcplb/ is Facebook's DHCP server - internally it uses https://github.com/insomniacslk/dhcp the dhcp library makes it pretty easy to write a server handler that will just return a hardcoded ip and path to the bootfile. see the server6 example for a very simplified demo of the interface you need to implement here is an example for how to write a tftp server module using FBTFTP https://github.com/facebook/fbtftp/blob/master/examples/server.py

some maybe-helfpul rfcs: https://datatracker.ietf.org/doc/html/rfc5970 (network booting with dhcpv6) https://datatracker.ietf.org/doc/html/rfc3315 (dhcpv6 in general)

[baioc]

@vmagro QEMU has built-in DHCP and TFTP servers, but they require using -netdev user (which uses SLIRP) instead of tap.

[vmagro]

This is the initrd that I mentioned, you can use the vmlinuz from the fedora kernel we make available and this initrd as your pxelinux payload. This initrd will boot to a userspace that has all the dependencies required for vmtest https://github.com/facebookincubator/antlir/blob/f5a42c2b60cae3ff98dab38a2ed4c5bbe815ab8c/antlir/linux/initrd/vmtest/BUCK#L114

[baioc]

Just to put some info here for the next person who will tackle this:

1. Figure out whether or not rebuilding an iPXE rom is needed. It seems like the one bundled in QEMU has IPv6 disabled.
2. The tap network is created after sidecars are launched, so we need to change that order to be able to create the DHCPv6 and TFTP servers and have them listen in that network.
3. Facebook's dhcplb seems overkill for this, just dhcp has everything we need.
4. However, we currently have no wrappers around Buck rules for Go binaries/libraries. But, since sidecars run on the host, all you need is a compiled executable on some known path.
5. The TFTP server should have a pxelinux image. This probably needs to be added as a third-party dependency. The bare minimum needed are the files pxelinux.0 and ldlinux.c32, which should be on the same directory.
6. Create a PXE configuration file in the TFTP server, either a default or one associated to the VM's address. That configuration can point to the Fedora kernel and initrd we already use in Antlir, which must also be copied into the TFTP server.

I found this network infrastructure quite hard to test on a local machine, but eventually figured out QEMU wasn't sending DHCPv6 requests, thus the item 1 on the list above. In order to test just the TFTP and PXE setup, one can use QEMU's builtin stack (uses DHCP v4):

$ qemu-system-x86_64 \
    -netdev user,id=net0,tftp=path/to/tftp,bootfile=pxelinux.0 \
    -device virtio-net-pci,netdev=net0,mac=00:00:00:00:00:01