search

openshift / os

89 stars 104 forks source link

[4.13-9.2/ppc64le] cosa buildextend-live fails on mksquashfs #1157

Closed travier closed 1 year ago

travier commented 1 year ago 
[2023-02-03T15:41:53.509Z] + cosa buildextend-live
[2023-02-03T15:41:53.509Z] time="2023-02-03T15:41:53Z" level=warning msg="please create a known_hosts file. The next time this host is connected to, podman will add it to known_hosts"
[2023-02-03T15:41:53.762Z] time="2023-02-03T15:41:53Z" level=warning msg="please create a known_hosts file. The next time this host is connected to, podman will add it to known_hosts"
[2023-02-03T15:41:54.681Z] Targeting build: 413.92.202302031414-0
[2023-02-03T15:41:54.681Z] 2023-02-03 15:41:54,647 INFO - Running command: ['/usr/bin/ostree', 'ls', '--repo', '/home/jenkins/agent/workspace/build-arch/tmp/repo', '--nul-filenames-only', '4bd3384e967bd2ba51c27b197da4b4e8e40584f0f9519b74fdeb8fb2da4baf82', '/usr/lib/modules']
[2023-02-03T15:41:54.681Z] 2023-02-03 15:41:54,663 INFO - Running command: ['/usr/bin/ostree', 'checkout', '--force-copy', '--repo', '/home/jenkins/agent/workspace/build-arch/tmp/repo', '--user-mode', '--subpath', '/usr/lib/modules/5.14.0-247.el9.ppc64le/vmlinuz', '4bd3384e967bd2ba51c27b197da4b4e8e40584f0f9519b74fdeb8fb2da4baf82', '/home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/live/images/pxeboot']
[2023-02-03T15:41:54.938Z] 2023-02-03 15:41:54,903 INFO - Running command: ['/usr/bin/ostree', 'checkout', '--force-copy', '--repo', '/home/jenkins/agent/workspace/build-arch/tmp/repo', '--user-mode', '--subpath', '/usr/lib/modules/5.14.0-247.el9.ppc64le/initramfs.img', '4bd3384e967bd2ba51c27b197da4b4e8e40584f0f9519b74fdeb8fb2da4baf82', '/home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/live/images/pxeboot']
[2023-02-03T15:41:55.192Z] 2023-02-03 15:41:55,033 INFO - Running command: ['/usr/bin/ostree', 'cat', '--repo', '/home/jenkins/agent/workspace/build-arch/tmp/repo', '4bd3384e967bd2ba51c27b197da4b4e8e40584f0f9519b74fdeb8fb2da4baf82', '/usr/share/coreos-installer/example-config.yaml']
[2023-02-03T15:41:55.192Z] 2023-02-03 15:41:55,070 INFO - Running command: ['/usr/bin/ostree', 'checkout', '--repo', '/home/jenkins/agent/workspace/build-arch/tmp/repo', '--user-mode', '--subpath', '/usr/lib/os-release', '4bd3384e967bd2ba51c27b197da4b4e8e40584f0f9519b74fdeb8fb2da4baf82', '/tmp/tmp77acu3n8']
[2023-02-03T15:41:56.111Z] Generating osmet file for 512b metal image
[2023-02-03T15:41:56.111Z] 2023-02-03 15:41:55,834 INFO - Running command: ['/usr/lib/coreos-assembler/runvm-coreos-installer', 'builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal.ppc64le.raw', '/home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/rhcos-413.92.202302031414-0-metal.ppc64le.raw.osmet', 'pack', 'osmet', '/dev/disk/by-id/virtio-coreos', '--description', 'CentOS Stream CoreOS 413.92.202302031414-0 (Plow)', '--checksum', 'ba3669cb89ad2fc04bec03efcce2d2ab878355571b7f713752b38b76781649e6', '--output', '/var/tmp/coreos-installer-output']
[2023-02-03T15:42:17.993Z] + RUST_BACKTRACE=full
[2023-02-03T15:42:17.993Z] + chroot /sysroot/ostree/deploy/rhcos/deploy/4bd3384e967bd2ba51c27b197da4b4e8e40584f0f9519b74fdeb8fb2da4baf82.0 env -C /home/jenkins/agent/workspace/build-arch coreos-installer pack osmet /dev/disk/by-id/virtio-coreos --description 'CentOS Stream CoreOS 413.92.202302031414-0 (Plow)' --checksum ba3669cb89ad2fc04bec03efcce2d2ab878355571b7f713752b38b76781649e6 --output /var/tmp/coreos-installer-output
[2023-02-03T15:42:18.246Z] Total OSTree objects scanned from /root: 18415 (18410 mapped, 5 empty)
[2023-02-03T15:42:18.246Z] Total OSTree objects found in /boot: 254
[2023-02-03T15:42:18.500Z] Duplicate extents dropped: 0
[2023-02-03T15:42:18.500Z] Overlapping extents clamped: 0
[2023-02-03T15:42:18.500Z] Total /root extents: 18411
[2023-02-03T15:42:18.500Z] Duplicate extents dropped: 0
[2023-02-03T15:42:18.500Z] Overlapping extents clamped: 0
[2023-02-03T15:42:18.500Z] Total /boot extents: 260
[2023-02-03T15:42:18.500Z] Packing image
[2023-02-03T15:44:39.841Z] Total bytes skipped: 2406667264
[2023-02-03T15:44:39.841Z] Total bytes written: 1373449216
[2023-02-03T15:44:39.841Z] Total bytes written (compressed): 7113912
[2023-02-03T15:44:39.841Z] Verifying that repacked image matches digest
[2023-02-03T15:45:06.299Z] Packing successful!
[2023-02-03T15:45:06.299Z] + '[' -f /var/tmp/coreos-installer-output ']'
[2023-02-03T15:45:06.299Z] + cp /var/tmp/coreos-installer-output /dev/virtio-ports/coreosout
[2023-02-03T15:45:06.299Z] Generating osmet file for 4k metal image
[2023-02-03T15:45:06.299Z] 2023-02-03 15:45:06,200 INFO - Running command: ['/usr/lib/coreos-assembler/runvm-coreos-installer', 'builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal4k.ppc64le.raw', '/home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/rhcos-413.92.202302031414-0-metal4k.ppc64le.raw.osmet', 'pack', 'osmet', '/dev/disk/by-id/virtio-coreos', '--description', 'CentOS Stream CoreOS 413.92.202302031414-0 (Plow)', '--checksum', 'b9b5db251ac4f85b8521bfc949c5ae09d4237240292c937a7f2431b36a10f8f8', '--output', '/var/tmp/coreos-installer-output']
[2023-02-03T15:45:32.753Z] + RUST_BACKTRACE=full
[2023-02-03T15:45:32.753Z] + chroot /sysroot/ostree/deploy/rhcos/deploy/4bd3384e967bd2ba51c27b197da4b4e8e40584f0f9519b74fdeb8fb2da4baf82.0 env -C /home/jenkins/agent/workspace/build-arch coreos-installer pack osmet /dev/disk/by-id/virtio-coreos --description 'CentOS Stream CoreOS 413.92.202302031414-0 (Plow)' --checksum b9b5db251ac4f85b8521bfc949c5ae09d4237240292c937a7f2431b36a10f8f8 --output /var/tmp/coreos-installer-output
[2023-02-03T15:45:32.753Z] Total OSTree objects scanned from /root: 18415 (18410 mapped, 5 empty)
[2023-02-03T15:45:32.753Z] Total OSTree objects found in /boot: 254
[2023-02-03T15:45:32.753Z] Duplicate extents dropped: 0
[2023-02-03T15:45:32.753Z] Overlapping extents clamped: 0
[2023-02-03T15:45:32.753Z] Total /root extents: 18410
[2023-02-03T15:45:32.753Z] Duplicate extents dropped: 0
[2023-02-03T15:45:32.753Z] Overlapping extents clamped: 0
[2023-02-03T15:45:32.753Z] Total /boot extents: 254
[2023-02-03T15:45:32.753Z] Packing image
[2023-02-03T15:47:39.131Z] Total bytes skipped: 2407038976
[2023-02-03T15:47:39.131Z] Total bytes written: 1373077504
[2023-02-03T15:47:39.131Z] Total bytes written (compressed): 7163196
[2023-02-03T15:47:39.131Z] Verifying that repacked image matches digest
[2023-02-03T15:48:17.769Z] Packing successful!
[2023-02-03T15:48:17.770Z] + '[' -f /var/tmp/coreos-installer-output ']'
[2023-02-03T15:48:17.770Z] + cp /var/tmp/coreos-installer-output /dev/virtio-ports/coreosout
[2023-02-03T15:48:17.770Z] Compressing squashfs with zstd
[2023-02-03T15:48:17.770Z] 2023-02-03 15:48:16,999 INFO - Running command: ['/usr/lib/coreos-assembler/gf-mksquashfs', 'builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal.ppc64le.raw', '/home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/root.squashfs', 'zstd']
[2023-02-03T15:48:17.770Z] +++ dirname /home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/root.squashfs
[2023-02-03T15:48:17.770Z] ++ mktemp -tdp /home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs gf-mksquashfs.XXXXXX
[2023-02-03T15:48:17.770Z] + tmpd=/home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.6GX6OJ
[2023-02-03T15:48:17.770Z] + tmp_dest=/home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.6GX6OJ/image.squashfs
[2023-02-03T15:48:17.770Z] + coreos_gf_run_mount builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal.ppc64le.raw --ro
[2023-02-03T15:48:17.770Z] + local mntarg=mount
[2023-02-03T15:48:17.770Z] + '[' builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal.ppc64le.raw = ro ']'
[2023-02-03T15:48:17.770Z] + coreos_gf_run builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal.ppc64le.raw --ro
[2023-02-03T15:48:17.770Z] + '[' -n '' ']'
[2023-02-03T15:48:17.770Z] + coreos_gf_launch builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal.ppc64le.raw --ro
[2023-02-03T15:48:17.770Z] + '[' -n '' ']'
[2023-02-03T15:48:17.770Z] ++ guestfish --listen -a builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal.ppc64le.raw --ro
[2023-02-03T15:48:17.770Z] + eval 'GUESTFISH_PID=26138; export GUESTFISH_PID'
[2023-02-03T15:48:17.770Z] ++ GUESTFISH_PID=26138
[2023-02-03T15:48:17.770Z] ++ export GUESTFISH_PID
[2023-02-03T15:48:17.770Z] + '[' -z 26138 ']'
[2023-02-03T15:48:17.770Z] ++ kola ncpu
[2023-02-03T15:48:17.770Z] + coreos_gf set-smp 128
[2023-02-03T15:48:17.770Z] + guestfish --remote -- set-smp 128
[2023-02-03T15:48:17.770Z] + coreos_gf run
[2023-02-03T15:48:17.770Z] + guestfish --remote -- run
[2023-02-03T15:48:29.930Z] + GUESTFISH_RUNNING=1
[2023-02-03T15:48:29.930Z] ++ coreos_gf findfs-label root
[2023-02-03T15:48:29.930Z] ++ guestfish --remote -- findfs-label root
[2023-02-03T15:48:29.930Z] + root=/dev/sda4
[2023-02-03T15:48:29.930Z] + coreos_gf mount /dev/sda4 /
[2023-02-03T15:48:29.930Z] + guestfish --remote -- mount /dev/sda4 /
[2023-02-03T15:48:29.930Z] + local boot
[2023-02-03T15:48:29.930Z] ++ coreos_gf findfs-label boot
[2023-02-03T15:48:29.930Z] ++ guestfish --remote -- findfs-label boot
[2023-02-03T15:48:29.930Z] + boot=/dev/sda3
[2023-02-03T15:48:29.930Z] + coreos_gf mount /dev/sda3 /boot
[2023-02-03T15:48:29.930Z] + guestfish --remote -- mount /dev/sda3 /boot
[2023-02-03T15:48:29.930Z] + local partitions
[2023-02-03T15:48:29.930Z] + local label
[2023-02-03T15:48:29.930Z] ++ coreos_gf list-partitions
[2023-02-03T15:48:29.930Z] ++ guestfish --remote -- list-partitions
[2023-02-03T15:48:29.930Z] + partitions='/dev/sda1
[2023-02-03T15:48:29.930Z] /dev/sda2
[2023-02-03T15:48:29.930Z] /dev/sda3
[2023-02-03T15:48:29.930Z] /dev/sda4'
[2023-02-03T15:48:29.930Z] + for pt in $partitions
[2023-02-03T15:48:29.930Z] ++ coreos_gf vfs-label /dev/sda1
[2023-02-03T15:48:29.930Z] ++ guestfish --remote -- vfs-label /dev/sda1
[2023-02-03T15:48:29.930Z] + label=
[2023-02-03T15:48:29.930Z] + '[' '' == EFI-SYSTEM ']'
[2023-02-03T15:48:29.930Z] + for pt in $partitions
[2023-02-03T15:48:29.930Z] ++ coreos_gf vfs-label /dev/sda2
[2023-02-03T15:48:29.930Z] ++ guestfish --remote -- vfs-label /dev/sda2
[2023-02-03T15:48:29.930Z] + label=
[2023-02-03T15:48:29.930Z] + '[' '' == EFI-SYSTEM ']'
[2023-02-03T15:48:29.930Z] + for pt in $partitions
[2023-02-03T15:48:29.930Z] ++ coreos_gf vfs-label /dev/sda3
[2023-02-03T15:48:29.930Z] ++ guestfish --remote -- vfs-label /dev/sda3
[2023-02-03T15:48:29.930Z] + label=boot
[2023-02-03T15:48:29.930Z] + '[' boot == EFI-SYSTEM ']'
[2023-02-03T15:48:29.930Z] + for pt in $partitions
[2023-02-03T15:48:29.930Z] ++ coreos_gf vfs-label /dev/sda4
[2023-02-03T15:48:29.930Z] ++ guestfish --remote -- vfs-label /dev/sda4
[2023-02-03T15:48:29.930Z] + label=root
[2023-02-03T15:48:29.930Z] + '[' root == EFI-SYSTEM ']'
[2023-02-03T15:48:29.930Z] ++ coreos_gf ls /ostree/deploy
[2023-02-03T15:48:29.930Z] ++ guestfish --remote -- ls /ostree/deploy
[2023-02-03T15:48:29.930Z] + stateroot=/ostree/deploy/rhcos
[2023-02-03T15:48:29.930Z] ++ coreos_gf ls /ostree/deploy/rhcos/deploy
[2023-02-03T15:48:29.930Z] ++ guestfish --remote -- ls /ostree/deploy/rhcos/deploy
[2023-02-03T15:48:29.930Z] ++ grep -v .origin
[2023-02-03T15:48:29.930Z] + deploydir=/ostree/deploy/rhcos/deploy/4bd3384e967bd2ba51c27b197da4b4e8e40584f0f9519b74fdeb8fb2da4baf82.0
[2023-02-03T15:48:29.930Z] + export stateroot deploydir
[2023-02-03T15:48:29.930Z] + coreos_gf download /ostree/repo/config /home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.6GX6OJ/config
[2023-02-03T15:48:29.930Z] + guestfish --remote -- download /ostree/repo/config /home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.6GX6OJ/config
[2023-02-03T15:48:29.930Z] + grep -v readonly=true /home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.6GX6OJ/config
[2023-02-03T15:48:29.931Z] + coreos_gf upload /home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.6GX6OJ/config.new /ostree/repo/config
[2023-02-03T15:48:29.931Z] + guestfish --remote -- upload /home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.6GX6OJ/config.new /ostree/repo/config
[2023-02-03T15:48:29.931Z] + coreos_gf mksquashfs / /home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.6GX6OJ/image.squashfs compress:zstd
[2023-02-03T15:48:29.931Z] + guestfish --remote -- mksquashfs / /home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.6GX6OJ/image.squashfs compress:zstd
[2023-02-03T15:48:36.442Z] libguestfs: error: mksquashfs: /: 
[2023-02-03T15:48:36.442Z] + _coreos_gf_cleanup
[2023-02-03T15:48:36.442Z] + guestfish --remote -- exit
[2023-02-03T15:48:36.442Z] + :
[2023-02-03T15:48:36.442Z] 2023-02-03 15:48:35,900 ERROR - Command returned bad exitcode
[2023-02-03T15:48:36.442Z] 2023-02-03 15:48:35,900 ERROR - COMMAND: ['/usr/lib/coreos-assembler/gf-mksquashfs', 'builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal.ppc64le.raw', '/home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/root.squashfs', 'zstd']
[2023-02-03T15:48:36.442Z] Traceback (most recent call last):
[2023-02-03T15:48:36.442Z]   File "/usr/lib/coreos-assembler/cmd-buildextend-live", line 736, in <module>
[2023-02-03T15:48:36.442Z]     generate_iso()
[2023-02-03T15:48:36.442Z]   File "/usr/lib/coreos-assembler/cmd-buildextend-live", line 331, in generate_iso
[2023-02-03T15:48:36.442Z]     runcmd(['/usr/lib/coreos-assembler/gf-mksquashfs',
[2023-02-03T15:48:36.442Z]   File "/usr/lib/coreos-assembler/cosalib/cmdlib.py", line 78, in runcmd
[2023-02-03T15:48:36.442Z]     raise e
[2023-02-03T15:48:36.442Z]   File "/usr/lib/coreos-assembler/cosalib/cmdlib.py", line 70, in runcmd
[2023-02-03T15:48:36.442Z]     cp = subprocess.run(cmd, **pargs)
[2023-02-03T15:48:36.442Z]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-02-03T15:48:36.442Z]   File "/usr/lib64/python3.11/subprocess.py", line 571, in run
[2023-02-03T15:48:36.442Z]     raise CalledProcessError(retcode, process.args,
[2023-02-03T15:48:36.442Z] subprocess.CalledProcessError: Command '['/usr/lib/coreos-assembler/gf-mksquashfs', 'builds/413.92.202302031414-0/ppc64le/rhcos-413.92.202302031414-0-metal.ppc64le.raw', '/home/jenkins/agent/workspace/build-arch/tmp/buildpost-live/initrd-rootfs/root.squashfs', 'zstd']' returned non-zero exit status 1.
[2023-02-03T15:48:36.442Z] error: failed to execute cmd-buildextend-live: exit status 1
ravanelli commented 1 year ago

I'm also getting the same issue for 4.12 for RHEL9

jlebon commented 1 year ago

Can you show the output of the command with LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 set in the environment?