Using docker-qemu for CI testing

[aparcar]

Maybe this is very naughty, but I'd like to do some CI tests with OpenWrt and it's firewall. While OpenWrt is already ported to Docker, playing with iptables would require additional capabilities for the CI executor. I came up with the idea of running OpenWrt as a qemu VM within a docker container ( :boom: ). This would emulate hardware and therefore could run "privileged" on any docker CI system.

A small PoC already fires up the OpenWrt images, however then does not receive any further commands from the CI. Would that be possible and I'm just missing a parameter?

[tianon]

On the contrary, I think this is a really, really cool idea. :smile:

I think the problem with your PoC is that start-qemu -nographic is going to run until the VM shuts down again.

You might need to build yourself an image FROM this one that embeds that system image directly so you can run it as a "service" in the CI configuration that your CI steps then interact with externally (via SSH or something similar).

Also note that without --device /dev/kvm, this is going to have a bit of a performance problem (since it will be emulating a CPU instead of doing virtualization), so you might not be able to get things to start/run in a reasonable amount of time on public CI infrastructure (although OpenWrt is pretty minimal and likely boots fast, so perhaps one of the few projects where this is more feasible).

[aparcar]

You might need to build yourself an image FROM this one that embeds that system image directly so you can run it as a "service" in the CI configuration that your CI steps then interact with externally (via SSH or something similar).

Interesting idea, will look into that. However testing (and breaking) the firewall over SSH sounds like a bad idea. I'd need something like a serial console or something which is independent of network.

Also note that without --device /dev/kvm, this is going to have a bit of a performance problem (since it will be emulating a CPU instead of doing virtualization), so you might not be able to get things to start/run in a reasonable amount of time on public CI infrastructure (although OpenWrt is pretty minimal and likely boots fast, so perhaps one of the few projects where this is more feasible).

Adding kvm should be possible, thanks for the advise.

[aparcar]

@tianon Hi, I've been testing for some time with partial luck! Please see a current PoC here. The service idea works surprisingly good, however OpenWrt comes per default with a firewall blocking SSH over the primary interface (wan access). Modifications before booting would be possible but somewhat dirty, also the "break the firewall testing" exists.

Do you see any chance to fire up qemu and let the CI detect that there is a stdin available again, allowing to pass command through the docker container directly into qemu?

[tianon]

I don't know of any CI that would do so "automatically" (you're essentially asking it to detect that your running command has now created a new command prompt). I think your best bet for that would be something with docker run -d and clever use of docker attach.

[aparcar]

You got an idea on how to attach to the running process? That seem to involve additional qemu magic.

[tianon]

I mean use serial console with stdio and docker attach to connect to the running container afterwards to inject input.