Failure when trying to run command zypper in system-root via chroot

[thimslugga]

SLES4SAP 12 SP5 (BYOS) -> 15 SP1 dist migration. Base OS was fully patched after launching. The dist migration did complete the upgrade to 15 SP1 despite the error. However this error was logged very early and causes end users to waste time triaging.

• ami-0a00940dcdcc6c61a
• suse-sles-sap-12-sp5-byos-v20240812-hvm-ssd-x86_64-b55a05e8-4f51-4d96-9a7f-3cf1dc7b12b4

Name           : SLES15-Migration
Version        : 2.0.37-6

Name           : suse-migration-sle15-activation
Version        : 2.0.37-6.41.6

Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: Installing the shim package
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: Running chroot /system-root zypper in shim
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: Calling: ['chroot', '/system-root', 'zypper', 'in', 'shim']
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: EXEC: Failed with stderr: chroot: failed to run command ‘zypper’: No such file or directory
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: , stdout: (no output on stdout)
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: Failed to install the shim package: chroot: stderr: chroot: failed to run command ‘zypper’: No such file or directory
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: , stdout: (no output on stdout)
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: Traceback (most recent call last):
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:   File "/usr/lib/python3.6/site-packages/suse_migration_services/units/update_bootloader.py", line 64, in install_shim_package
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:     'shim'
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:   File "/usr/lib/python3.6/site-packages/suse_migration_services/command.py", line 116, in run
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:     command[0], error.decode(), output.decode()
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: suse_migration_services.exceptions.DistMigrationCommandException: chroot: stderr: chroot: failed to run command ‘zypper’: No such file or directory
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: , stdout: (no output on stdout)
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: The above exception was the direct cause of the following exception:
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: Traceback (most recent call last):
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:   File "/usr/bin/suse-migration-update-bootloader", line 11, in <module>
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:     load_entry_point('suse-migration-services==2.0.37', 'console_scripts', 'suse-migration-update-bootloader')()
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:   File "/usr/lib/python3.6/site-packages/suse_migration_services/units/update_bootloader.py", line 42, in main
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:     install_shim_package(root_path)
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:   File "/usr/lib/python3.6/site-packages/suse_migration_services/units/update_bootloader.py", line 75, in install_shim_package
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]:     ) from issue
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: suse_migration_services.exceptions.DistMigrationCommandException: Failed to install the shim package: chroot: stderr: chroot: failed to run command ‘zypper’: No such file or directory
Sep 04 14:19:47 localhost suse-migration-update-bootloader[772]: , stdout: (no output on stdout)
Sep 04 14:19:47 localhost systemd[1]: suse-migration-update-bootloader.service: Main process exited, code=exited, status=1/FAILURE
Sep 04 14:19:47 localhost systemd[1]: Failed to start Update the bootloader.
Sep 04 14:19:47 localhost systemd[1]: suse-migration-update-bootloader.service: Unit entered failed state.
Sep 04 14:19:47 localhost systemd[1]: suse-migration-update-bootloader.service: Failed with result 'exit-code'.
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['mkdir', '-p', '/system-root']
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Running mount system service
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Checking /system-root is mounted
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Mount system service: /system-root is mounted
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Checking /run/initramfs/isoscan is mounted
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Mount system service: /run/initramfs/isoscan is mounted
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['mount', '-o', 'remount,rw', '/run/initramfs/isoscan']
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Reading fstab from associated disks
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['lsblk', '-p', '-n', '-r', '-o', 'NAME,TYPE']
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Lookup for fstab on /dev/nvme0n1p1 ...
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['mount', '/dev/nvme0n1p1', '/system-root']
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: EXEC: Failed with stderr: mount: /system-root: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error.
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: , stdout: (no output on stdout)
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Exception on mount of /dev/nvme0n1p1: mount: stderr: mount: /system-root: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error.
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: , stdout: (no output on stdout)
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Lookup for fstab on /dev/nvme0n1p2 ...
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['mount', '/dev/nvme0n1p2', '/system-root']
Sep 04 14:19:47 localhost kernel: FAT-fs (nvme0n1p2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: No /dev/nvme0n1p2 found on
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Umount /system-root
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['umount', '/system-root']
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Lookup for fstab on /dev/nvme0n1p3 ...
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['mount', '/dev/nvme0n1p3', '/system-root']
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Found /system-root/etc/fstab on /dev/nvme0n1p3
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Mount system in /system-root
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Mounting /system-root/boot/efi
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['mount', '-o', 'defaults', '/dev/disk/by-label/EFI', '/system-root/boot/efi']
Sep 04 14:19:47 localhost kernel: FAT-fs (nvme0n1p2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Mounting kernel file systems inside /system-root
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['mount', '-t', 'devtmpfs', 'devtmpfs', '/system-root/dev']
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['mount', '-t', 'proc', 'proc', '/system-root/proc']
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: Calling: ['mount', '-t', 'sysfs', 'sysfs', '/system-root/sys']
Sep 04 14:19:47 localhost suse-migration-mount-system[773]: The migration file '/etc/migration-config.yml' has been updated with '/system-root/etc/sle-migration-service.yml' info

[rtamalin]

I believe that this issue is likely related to the issue fixed by #283 - the suse-migration-update-bootloader service incorrectly raced with the suse-migration-mount-system service and attempted to run commands under the /system-root chroot area before the appropriate file systems had been mounted.

The fix of the issue is included in the v2.0.40 release, which we are currently working on getting released through the official SUSE product release channels.

[thimslugga]

@rtamalin yes, it definitely looks like it's related to that PR.

With SLES 12 EoM quickly approaching, I am seeing more and more dist migrations happening so getting the new release will help reduce escalations for this red herring.

[schaefi]

v2.0.40 release is done and I also agree with @rtamalin that this issue is related to the systemd services order which was fixed by Fergel. I'm closing this one. If you believe this is not fixed please re-open.

Thanks