`virtiofs` doesn't work on `s390x`

[simondeziel]

In a s390x VM, trying to mount the config drive fails:

# uname -m
s390x
# lxc launch ubuntu-daily:22.04 v1 --vm
Creating v1
Starting v1
# sleep 30
# lxc exec v1 -- mount -t virtiofs config /srv
mount: /srv: wrong fs type, bad option, bad superblock on config, missing codepage or helper program, or other error.

LXD logs contain:

level=warning msg="Unable to use virtio-fs for config drive, using 9p as a fallback" err="Architecture unsupported"

Looking at https://github.com/canonical/lxd/blob/main/lxd/instance/drivers/driver_qemu.go#L1358, failure to start virtiofsd seems to be why the qemu config does not have the dev-qemu_config-drive-virtio-fs device one would expect:

# grep -E '(^type|config-drive)' /var/snap/lxd/common/lxd/logs/v1/qemu.conf
type = "s390-ccw-virtio"
[device "dev-qemu_config-drive-9p"]

While on amd64 it's present:

# grep -E '(^type|config-drive)' /var/snap/lxd/common/lxd/logs/v2/qemu.conf
type = "q35"
type = "node"
[device "dev-qemu_config-drive-9p"]
[device "dev-qemu_config-drive-virtio-fs"]

but the snap has the binary built for it and has no apparent problem running this simple version check:

# LD_LIBRARY_PATH=/snap/lxd/current/lib/:/snap/lxd/current/lib/x86_64-linux-gnu/ nsenter --mount=/run/snapd/ns/lxd.mnt /snap/lxd/current/bin/virtiofsd --version
virtiofsd 1.8.0

Additional information:

This affects both latest/edge and 5.0/edge:

# snap list lxd
Name  Version      Rev    Tracking     Publisher   Notes
lxd   git-00ab49c  26578  latest/edge  canonical✓  -

# snap list lxd
Name  Version      Rev    Tracking  Publisher   Notes
lxd   git-ccdba5c  26493  5.0/edge  canonical✓  -

[tomponline]

@simondeziel this is due to this section:

https://github.com/canonical/lxd/blob/main/lxd/device/device_utils_disk.go#L454-L458

I've tested on our s390x machine (which runs Focal) and although the virtiofsd daemon starts on the host, there is some issue in the guest (running Jammy) that means the agent doesn't start properly (likely because it cannot mount the config drive).

So I'm happy that this isn't a LXD 5.0.3 or LXD 24.04 release blocker.

Although it would be interesting to speak to IS about getting our s390x system upgraded to Jammy and see if its still an issue so we can remove the restriction in 6.0.

[simondeziel]

@tomponline I made the request to get us a few LPARs running Jammy and tentatively Noble. Depending on how it goes, should we remove virtiofsd from the non-amd64 snap?