search

suse-edge / edge-image-builder

Tool for creating and configuring a set of images to automate the deployment of Edge environments
Apache License 2.0
33 stars 20 forks source link

Doesn't seem to work on aarch64 #68

Closed e-minguez closed 5 months ago

e-minguez commented 6 months ago

Trying to build a raw image on an openSUSE box on an aarch64 VM on top of OSX doesn't seem to work.

After enabling debugging this is the error log:

eminguez@lima-default:~> podman run --rm -it -e LIBGUESTFS_DEBUG=1 -e LIBGUESTFS_TRACE=1 -v $(pwd)/eib:/eib eib:dev /bin/eib -config-file config-eib.yaml -config-dir /eib -build-dir /eib/_build/
Generating image customization components...
Identifier ................... [SUCCESS]
Custom Files ................. [SKIPPED]
Network ...................... [SKIPPED]
Users ........................ [SUCCESS]
Rpm .......................... [SKIPPED]
Systemd ...................... [SKIPPED]
Elemental .................... [SKIPPED]
Suma ......................... [SKIPPED]
Building RAW image...
Kernel Params ................ [SUCCESS]
Error building RAW image, check the logs under the build directory for more information.

eminguez@lima-default:~> cat eib/_build/eib-build-Dec20_12-49-23.log
2023-12-20T12:49:23.450Z    INFO    combustion/network.go:51    Configuring network component...
2023-12-20T12:49:23.450Z    INFO    combustion/network.go:55    Skipping network component. Configuration is not provided
2023-12-20T12:49:23.450Z    INFO    combustion/elemental.go:30  Skipping elemental registration component. Configuration is not provided
2023-12-20T12:49:26.175Z    ERROR   build/raw.go:44 Failed to build raw image   {"error": "libguestfs: trace: set_verbose true\nlibguestfs: trace: set_verbose = 0\nlibguestfs: create: flags = 0, handle = 0xaaaad9cabf40, program = guestfish\nlibguestfs: trace: add_drive \"/eib/SLEMicro.x86_64--5.5.0-Default-GM-userpass-openstack.raw\"\nlibguestfs: trace: add_drive = 0\nlibguestfs: trace: is_config\nlibguestfs: trace: is_config = 1\nlibguestfs: trace: launch\nlibguestfs: trace: max_disks\nlibguestfs: trace: max_disks = 255\nlibguestfs: trace: get_tmpdir\nlibguestfs: trace: get_tmpdir = \"/tmp\"\nlibguestfs: trace: version\nlibguestfs: trace: version = <struct guestfs_version = major: 1, minor: 48, release: 6, extra: , >\nlibguestfs: trace: get_backend\nlibguestfs: trace: get_backend = \"direct\"\nlibguestfs: launch: program=guestfish\nlibguestfs: launch: version=1.48.6\nlibguestfs: launch: backend registered: libvirt\nlibguestfs: launch: backend registered: direct\nlibguestfs: launch: backend=direct\nlibguestfs: launch: tmpdir=/tmp/libguestfsq5L96d\nlibguestfs: launch: umask=0022\nlibguestfs: launch: euid=0\nlibguestfs: trace: get_cachedir\nlibguestfs: trace: get_cachedir = \"/var/tmp\"\nlibguestfs: begin building supermin appliance\nlibguestfs: run supermin\nlibguestfs: command: run: /usr/bin/supermin\nlibguestfs: command: run: \\ --build\nlibguestfs: command: run: \\ --verbose\nlibguestfs: command: run: \\ --if-newer\nlibguestfs: command: run: \\ --lock /var/tmp/.guestfs-0/lock\nlibguestfs: command: run: \\ --copy-kernel\nlibguestfs: command: run: \\ -f ext2\nlibguestfs: command: run: \\ --host-cpu aarch64\nlibguestfs: command: run: \\ /usr/lib64/guestfs/supermin.d\nlibguestfs: command: run: \\ -o /var/tmp/.guestfs-0/appliance.d\nsupermin: version: 5.2.2\nsupermin: rpm: detected RPM version 4.14\nsupermin: rpm: detected RPM architecture aarch64\nsupermin: rpm: detected zypper version 1.14.66\nsupermin: package handler: opensuse/rpm\nsupermin: acquiring lock on /var/tmp/.guestfs-0/lock\nsupermin: build: /usr/lib64/guestfs/supermin.d\nsupermin: reading the supermin appliance\nsupermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)\nsupermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)\nsupermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles\nsupermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles\nsupermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)\nsupermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages\nsupermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)\nsupermin: build: visiting /usr/lib64/guestfs/supermin.d/zz-scripts.tar.gz type gzip base image (tar)\nsupermin: mapping package names to installed packages\nsupermin: resolving full list of package dependencies\nsupermin: build: 133 packages, including dependencies\nsupermin: build: 6092 files\nsupermin: build: 4306 files, after matching excludefiles\nsupermin: build: 4307 files, after adding hostfiles\nsupermin: build: 4298 files, after removing unreadable files\nsupermin: build: 4305 files, after munging\nsupermin: kernel: looking for kernel using environment variables ...\nsupermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...\nsupermin: kernel: looking for kernels in /boot ...\nsupermin: kernel: kernel version of /boot/vmlinux-5.14.21-150500.55.39-default.gz = 5.14.21-150500.55.39-default (from filename)\nsupermin: kernel: picked modules path /lib/modules/5.14.21-150500.55.39-default\nsupermin: kernel: kernel version of /boot/Image-5.14.21-150500.55.39-default = 5.14.21-150500.55.39-default (from filename)\nsupermin: kernel: picked modules path /lib/modules/5.14.21-150500.55.39-default\nsupermin: kernel: picked vmlinuz /boot/vmlinux-5.14.21-150500.55.39-default.gz\nsupermin: kernel: kernel_version 5.14.21-150500.55.39-default\nsupermin: kernel: modpath /lib/modules/5.14.21-150500.55.39-default\nsupermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.uw6rgdws/root'\nsupermin: ext2: populating from base image\nsupermin: ext2: copying files from host filesystem\nsupermin: ext2: copying kernel modules\nsupermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.uw6rgdws/initrd'\nsupermin: ext2: wrote 44 modules to minimal initrd\nsupermin: renaming /var/tmp/.guestfs-0/appliance.d.uw6rgdws to /var/tmp/.guestfs-0/appliance.d\nlibguestfs: finished building supermin appliance\nlibguestfs: begin testing qemu features\nlibguestfs: trace: get_cachedir\nlibguestfs: trace: get_cachedir = \"/var/tmp\"\nlibguestfs: checking for previously cached test results of /usr/bin/qemu-system-aarch64, in /var/tmp/.guestfs-0\nlibguestfs: command: run: /usr/bin/qemu-system-aarch64\nlibguestfs: command: run: \\ -display none\nlibguestfs: command: run: \\ -help\nlibguestfs: command: run: /usr/bin/qemu-system-aarch64\nlibguestfs: command: run: \\ -display none\nlibguestfs: command: run: \\ -machine virt,accel=kvm:tcg\nlibguestfs: command: run: \\ -device ?\nlibguestfs: command: run: echo '{ \"execute\": \"qmp_capabilities\" }' '{ \"execute\": \"query-qmp-schema\" }' '{ \"execute\": \"quit\" }'  | QEMU_AUDIO_DRV=none \"/usr/bin/qemu-system-aarch64\" -display none -machine \"virt,accel=kvm:tcg\" -qmp stdio\nlibguestfs: did not understand QMP monitor output from /usr/bin/qemu-system-aarch64 (ignored)\nlibguestfs: command: run: echo '{ \"execute\": \"qmp_capabilities\" }' '{ \"execute\": \"query-kvm\" }' '{ \"execute\": \"quit\" }'  | QEMU_AUDIO_DRV=none \"/usr/bin/qemu-system-aarch64\" -display none -machine \"virt,accel=kvm:tcg\" -qmp stdio\nlibguestfs: did not understand QMP monitor output from /usr/bin/qemu-system-aarch64 (ignored)\nlibguestfs: saving test results\nlibguestfs: QMP parse error: '[' or '{' expected near end of file (ignored)\nlibguestfs: QMP parse error: '[' or '{' expected near end of file (ignored)\nlibguestfs: qemu version: 7.1\nlibguestfs: qemu mandatory locking: yes\nlibguestfs: qemu KVM: enabled\nlibguestfs: trace: get_backend_setting \"force_tcg\"\nlibguestfs: trace: get_backend_setting = NULL (error)\nlibguestfs: trace: get_backend_setting \"force_kvm\"\nlibguestfs: trace: get_backend_setting = NULL (error)\nlibguestfs: trace: get_sockdir\nlibguestfs: trace: get_sockdir = \"/tmp\"\nlibguestfs: finished testing qemu features\nlibguestfs: trace: get_backend_setting \"gdb\"\nlibguestfs: trace: get_backend_setting = NULL (error)\n/usr/bin/qemu-system-aarch64 \\\n    -global virtio-blk-pci.scsi=off \\\n    -no-user-config \\\n    -nodefaults \\\n    -display none \\\n    -machine virt,gic-version=host,accel=kvm:tcg \\\n    -cpu host \\\n    -m 1280 \\\n    -no-reboot \\\n    -rtc driftfix=slew \\\n    -kernel /var/tmp/.guestfs-0/appliance.d/kernel \\\n    -initrd /var/tmp/.guestfs-0/appliance.d/initrd \\\n    -object rng-random,filename=/dev/urandom,id=rng0 \\\n    -device virtio-rng-pci,rng=rng0 \\\n    -device virtio-scsi-pci,id=scsi \\\n    -drive file=/eib/SLEMicro.x86_64--5.5.0-Default-GM-userpass-openstack.raw,cache=writeback,id=hd0,if=none \\\n    -device scsi-hd,drive=hd0 \\\n    -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none \\\n    -device scsi-hd,drive=appliance \\\n    -device virtio-serial-pci \\\n    -serial stdio \\\n    -chardev socket,path=/tmp/libguestfs3Brvgg/guestfsd.sock,id=channel0 \\\n    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \\\n    -append \"panic=1 console=ttyAMA0 earlyprintk=pl011,,0x9000000 ignore_loglevel efi-rtc=noprobe edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=fff998d0-2aaa-456a-81c8-4f00821731f5 selinux=0 guestfs_verbose=1 TERM=xterm\"\nWARNING: Image format was not specified for '/eib/SLEMicro.x86_64--5.5.0-Default-GM-userpass-openstack.raw' and probing guessed raw.\n         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.\n         Specify the 'raw' format explicitly to remove the restrictions.\nCould not access KVM kernel module: No such file or directory\nqemu-system-aarch64: failed to initialize kvm: No such file or directory\nqemu-system-aarch64: falling back to tcg\nqemu-system-aarch64: gic-version=host requires KVM\nlibguestfs: error: appliance closed the connection unexpectedly, see earlier error messages\nlibguestfs: child_cleanup: 0xaaaad9cabf40: child process died\nlibguestfs: sending SIGTERM to process 168\nlibguestfs: error: /usr/bin/qemu-system-aarch64 exited with error status 1, see debug messages above\nlibguestfs: error: guestfs_launch failed, see earlier error messages\nlibguestfs: trace: launch = -1 (error)\nlibguestfs: trace: close\nlibguestfs: closing guestfs handle 0xaaaad9cabf40 (state 0)\nlibguestfs: command: run: rm\nlibguestfs: command: run: \\ -rf /tmp/libguestfsq5L96d\nlibguestfs: command: run: rm\nlibguestfs: command: run: \\ -rf /tmp/libguestfs3Brvgg\n"}
github.com/suse-edge/edge-image-builder/pkg/build.(*Builder).buildRawImage
    /src/pkg/build/raw.go:44
github.com/suse-edge/edge-image-builder/pkg/build.(*Builder).Build
    /src/pkg/build/build.go:42
main.main
    /src/cmd/eib/main.go:119
runtime.main
    /usr/lib64/go/1.21/src/runtime/proc.go:267
2023-12-20T12:49:26.175Z    FATAL   eib/main.go:120 An error occurred building the image    {"error": "running the image modification script: exit status 1"}
main.main
    /src/cmd/eib/main.go:120
runtime.main
    /usr/lib64/go/1.21/src/runtime/proc.go:267
atanasdinov commented 6 months ago 
Could not access KVM kernel module: No such file or directory
qemu-system-aarch64: failed to initialize kvm: No such file or directory
qemu-system-aarch64: falling back to tcg
qemu-system-aarch64: gic-version=host requires KVM

Not really familiar with these sort of issues but I couldn't find a qemu-kvm package for aarch64 systems.

CC: @jdob @agracey

jdob commented 6 months ago

Pretty sure these are the same errors I'd been seeing for a while. It seems like the packaging is different between x86 and arch64, so the packages we install on x86 have everything we need to run but on arch64 there are either different or extra packages that are needed. I didn't dig any further.

To answer yesterday's question, I remember what the setup was when the raw image stuff was first added:

agracey commented 6 months ago

When I do the build in OBS, I can build both x86 and aarch64 images that include the correct packages for their respective architecture. Let's see if that resolves this issue... I'm not completely sure that it will but it might give more "interesting" errors 🤣

jdob commented 5 months ago

We are not supporting aarch64 for the initial release. Closing this bug since we'll encounter this (and will likely encounter a ton of other issues when we investigate formally adding it).