Testing xen-arm-builder automatically

[lcdunstan]

I imagine that the current way that PRs for this project are tested is:

• Verify that the Travis build was successful.
• Manually clone the PR to a developer's PC.
• Start the Ubuntu-based privileged Docker container (possibly rebuild it first).
• Manually start the build, e.g. make clone && make build && make cubieboard2.img
• When it eventually finishes, exit the Docker container.
• Physically insert a microSD card into the PC and "dd" the image to it.
• Unplug the microSD card and plug it into a powered down cubieboard2.
• Optionally connect a serial cable to the console port.
• Power on the board. I assume that it is already connected to the LAN.
• If it boots OK then ssh cubieboard2.local (fiddling with known_hosts).
• Run some kind of manual test as a sanity check, like building and running a unikernel.
• Possibly repeat all steps for BOARD=cubietruck.

My proposal for automating this would be:

• A physical machine is designated as a build slave and is permanently connected to the serial console of a cubieboard2 and/or a cubietruck.
• A device allows the power to the board to be controlled by software.
• Automatically clone the PR, start the build container, build the image, and stop the container.
• Automatically power off the device, power it on, and simulate a keypress on the serial console to interrupt U-Boot.
• Send commands to U-Boot to download the built kernel, an initial ramdisk, and the built dtb to the board (in RAM only) via TFTP. No Xen yet...
• Boot the downloaded image into the initramfs.
• If everything worked OK, run tftp -g -r cubieboard2.img -l - 192.168.x.y | dd of=/dev/mmcblk0 bs=4096 (or equivalent) via the serial console.
• After it completes, power the board off and on and verify that the complete image has booted.
• Run some tests over SSH.

But do you consider xen-arm-builder to be important enough to warrant setting this up?

I've done some experiments including:

• Using a second cubieboard2 as the build machine, since it has suitable UARTs. -- You would know that the normal USB-to-serial device supplied with the cubieboard2 is not suitable for automation since it interferes with the ability to power off the board. That was why I tried out a second board initially. -- Builds are slow of course, but that could be done elsewhere, maybe even on Travis.
• Using one of these cheap wifi-controllable power plugs to control the power: http://hackaday.com/2014/11/13/hacking-a-20-wifi-smart-plug/ (configuring it can be painful but after that it's cool).
• Automation of most of the U-Boot commands can be done with pyboot -- Depends on conmux.

So, is this worthwhile?

[lcdunstan]

Here's a log of the build: https://jenkins-lukedunstan.rhcloud.com/job/xen-arm-builder_external/1/console

Here's a log of a script that writes the image to the sdcard and then boots it: https://jenkins-lukedunstan.rhcloud.com/job/load-image_external/4/console

(Jenkins is only used as a place to store log files)

[mor1]

Hi @infidel; As it happens, after the refresh we're starting to look at automating more of the build infrastructure to support better testing, including looking again at some of our local machine room infrastructure. Is this something you managed to push any further on in the meantime? It would be great to get something setup to test like this!

[lcdunstan]

I didn't spend much more time on it but it did work. Some shell scripts are at https://github.com/infidel/cubiebuild (jenkins-ssh.sh is the entry point) but it could potentially be much simpler, because the manner in which I used Jenkins is not normal, and the reason for that is because I was using a cubieboard2 as the build machine, installing the resulting image on a second cubieboard2.

It would be simpler to just have a proper server do the build and then have a serial device server (aka terminal server) to control the serial console of the target board, plus another network relay device to control the target power.

The basic concept is the same as what I did though: build an image file, power off the target board, power it on, send serial console commands to U-Boot to load the newly built kernel into RAM, then use tftp ... | dd ... to download the full image to SD card, then power cycle to boot it. The main missing piece is to run tests to verify that the new image is fully working.