Missing mount point for lava-overlay on device

[pawiecz]

My environment:

• Ubuntu 18.04
• Docker 19.03.13, build 4484c46d9d
• docker-compose 1.17.1 (prepackaged, from Ubuntu repo)

First health check job on a fresh lava-docker-compose'd instance fails on missing /lava-1/bin/lava-test-runner.

Tool is unavailable due to missing overlay at mount point /lava-1; job log:

/ # mount /dev/disk/by-uuid/fe280245-7699-4b7e-a3b6-cc62308db826 -t ext2 /lava-1
mount /dev/disk/by-uuid/fe280245-7699-4b7e-a3b6-cc62308db826 -t ext2 /lava-1
mount: mounting /dev/disk/by-uuid/fe280245-7699-4b7e-a3b6-cc62308db826 on /lava-1 failed: No such file or directory

I downloaded zImage and rootfs.cpio.gz, then extracted overlay from lava-dispatcher container during the job run:

docker cp lava_dispatcher:/var/lib/lava/dispatcher/tmp/1/apply-overlay-guest-1sldr0_0/lava-guest.qcow2 .

Then run device command directly on host:

/usr/bin/qemu-system-arm \
    -cpu cortex-a15 \
    -machine virt \
    -nographic \
    -net nic,model=virtio,macaddr=DE:AD:BE:EF:66:01 \
    -net tap \
    -m 1024 \
    -monitor none \
    -kernel zImage \
    -append "console=ttyAMA0,115200 root=/dev/ram0 debug verbose" \
    -initrd rootfs.cpio.gz \
    -drive format=qcow2,file=lava-guest.qcow2,media=disk,if=virtio,id=lavatest

After creating /lava-1 mount point manually overlay mounts successfully:

/ # ls -la /lava-1/bin
total 96
drwxr-xr-x    2 root     root          4096 Sep 23 17:42 .
drwxr-xr-x    7 root     root          4096 Sep 23 17:42 ..
-rwxr-xr-x    1 root     root           209 Sep 23 17:42 lava-add-keys
-rwxr-xr-x    1 root     root           264 Sep 23 17:42 lava-add-sources
-rwxr-xr-x    1 root     root           808 Sep 23 17:42 lava-background-process-start
-rwxr-xr-x    1 root     root           387 Sep 23 17:42 lava-background-process-stop
-rwxr-xr-x    1 root     root           111 Sep 23 17:42 lava-common-functions
-rwxr-xr-x    1 root     root           584 Sep 23 17:42 lava-echo-ipv4
-rwxr-xr-x    1 root     root           598 Sep 23 17:42 lava-install-packages
-rwxr-xr-x    1 root     root           239 Sep 23 17:42 lava-installed-packages
-rwxr-xr-x    1 root     root           601 Sep 23 17:42 lava-os-build
-rwxr-xr-x    1 root     root           169 Sep 23 17:42 lava-probe-channel
-rwxr-xr-x    1 root     root           157 Sep 23 17:42 lava-probe-ip
-rwxr-xr-x    1 root     root           146 Sep 23 17:42 lava-target-ip
-rwxr-xr-x    1 root     root           150 Sep 23 17:42 lava-target-mac
-rwxr-xr-x    1 root     root          1169 Sep 23 17:42 lava-target-storage
-rwxr-xr-x    1 root     root          2101 Sep 23 17:42 lava-test-case
-rwxr-xr-x    1 root     root           281 Sep 23 17:42 lava-test-event
-rwxr-xr-x    1 root     root           674 Sep 23 17:42 lava-test-feedback
-rwxr-xr-x    1 root     root           332 Sep 23 17:42 lava-test-raise
-rwxr-xr-x    1 root     root          1006 Sep 23 17:42 lava-test-reference
-rwxr-xr-x    1 root     root          1039 Sep 23 17:42 lava-test-runner
-rwxr-xr-x    1 root     root           631 Sep 23 17:42 lava-test-set
-rwxr-xr-x    1 root     root           210 Sep 23 17:42 lava-test-shell

Please let me know how can I mount overlay properly for the health check.

[pawiecz]

I managed to get health check running by downgrading LAVA Docker container images to 2019.05.

Version 2019.05 still used qemu 2.8.1 instead of 3.1.0 from LAVA 2019.12 (and later).

I plan to rebuild rootfs with more recent udev (220 in http://people.linaro.org/~dan.rue/health_checks/qemu/rootfs.cpio.gz) and I will post an update if it helps.

[pawiecz]

It turned out completely unrelated to rootfs or kernel in use. QEMU defaults changed in 3.x release (last one).