COSA fails on ppc64le/P9 with "fatal: Couldn'\''t find rc file, something went terribly wrong!"

[jcajka]

Bug Report

Environment

What operating system is being used to run coreos-assembler? Fedora 31 What operating system is being assembled? Fedora Is coreos-assembler running in Podman or Docker? podman If Podman, is coreos-assembler running privileged or unprivileged? privileged

Expected Behavior

Build succedes

Actual Behavior

.
.
.
Disk size estimated to 2849Mfatal: Couldn'\''t find rc file, something went terribly wrong!
++ ostree --repo=/srv/tmp/repo ls 2fa865b791e763deef7760b11bd5d8a80d420b3e5b3a17b367be5af120307755 /usr/lib/modules
++ grep -o '/usr/lib/modules/.*'
+ target_moduledir=/usr/lib/modules/5.4.8-200.fc31.ppc64le
+ ostree --repo=/srv/tmp/repo cat 2fa865b791e763deef7760b11bd5d8a80d420b3e5b3a17b367be5af120307755 /usr/lib/modules/5.4.8-200.fc31.ppc64le/config
+ grep -Eq '^CONFIG_FS_VERITY=y' tmp/target-kernel.config
+ disk_args+=("--boot-verity")
+ rm tmp/target-kernel.config
++ python3 -c 'import sys, yaml; args = yaml.safe_load(sys.stdin).get("extra-kargs", []); print(" ".join(args))'
+ kargs='mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0'
+ tty='console=tty0 console=hvc0,115200n8'
+ '[' ppc64le == s390x ']'
+ kargs='mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=hvc0,115200n8 ignition.platform.id=qemu'
++ python3 -c 'import sys, yaml; print(yaml.safe_load(sys.stdin).get("ostree-remote", "NONE"))'
+ ostree_remote=fedora
++ python3 -c 'import sys, yaml; print(yaml.safe_load(sys.stdin).get("save-var-subdirs-for-selabel-workaround", "NONE"))'
+ save_var_subdirs=NONE
++ python3 -c 'import sys, yaml; lf=yaml.safe_load(sys.stdin).get("luks_rootfs", ""); print("luks" if lf.lower() in ("yes", "true") else "")'
+ rootfs_type=
+ '[' -z '' ']'
++ python3 -c 'import sys, yaml; print(yaml.safe_load(sys.stdin).get("rootfs", "xfs"))'
+ rootfs_type=xfs
+ qemu-img create -f qcow2 /srv/tmp/build/fedora-coreos-31.20200110.dev.0-qemu.ppc64le.qcow2.tmp 8G
Formatting '/srv/tmp/build/fedora-coreos-31.20200110.dev.0-qemu.ppc64le.qcow2.tmp', fmt=qcow2 size=8589934592 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+ ref_arg=fedora/ppc64le/coreos/testing-devel
+ '[' -n '' ']'
+ target_drive=("-drive" "if=virtio,id=target,format=${image_format},file=${path}.tmp,cache=unsafe")
+ [[ qcow2 == raw ]]
+ runvm -drive if=virtio,id=target,format=qcow2,file=/srv/tmp/build/fedora-coreos-31.20200110.dev.0-qemu.ppc64le.qcow2.tmp,cache=unsafe -- /usr/lib/coreos-assembler/create_disk.sh --disk /dev/vda --buildid 31.20200110.dev.0 --imgid fedora-coreos-31.20200110.dev.0-qemu.ppc64le.qcow2 --grub-script /usr/lib/coreos-assembler/grub.cfg --kargs '"mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=hvc0,115200n8 ignition.platform.id=qemu"' --osname fedora-coreos --ostree-ref fedora/ppc64le/coreos/testing-devel --ostree-remote fedora --ostree-repo /srv/tmp/repo --save-var-subdirs NONE --rootfs-size 2336M --rootfs xfs --boot-verity
+ qemu_args=()
+ local qemu_args
+ true
+ case "$1" in
+ qemu_args+=("$1")
+ shift
+ true
+ case "$1" in
+ qemu_args+=("$1")
+ shift
+ true
+ case "$1" in
+ shift
+ break
+ local vmpreparedir=/srv/tmp/supermin.prepare
+ local vmbuilddir=/srv/tmp/supermin.build
+ rm -rf /srv/tmp/supermin.prepare /srv/tmp/supermin.build
+ mkdir -p /srv/tmp/supermin.prepare /srv/tmp/supermin.build
+ local rpms
+ '[' -n 1 ']'
+ filter='^#FEDORA '
+ '[' -n '' ']'
++ grep -v '^#'
++ sed 's/^#FEDORA //' /usr/lib/coreos-assembler/vmdeps.txt
+ rpms='
bash vim-minimal coreutils util-linux procps-ng kmod kernel-modules

rpm-ostree distribution-gpg-keys jq

systemd

dhcp-client bind-export-libs iproute

selinux-policy selinux-policy-targeted policycoreutils

python3 python3-gobject-base buildah podman skopeo iptables iptables-libs

cryptsetup

gdisk xfsprogs e2fsprogs dosfstools shim'
++ grep -v '^#'
+++ arch
++ sed 's/^#FEDORA //' /usr/lib/coreos-assembler/vmdeps-ppc64le.txt
+ archrpms='grub2 powerpc-utils'
+ supermin --prepare --use-installed -o /srv/tmp/supermin.prepare bash vim-minimal coreutils util-linux procps-ng kmod kernel-modules rpm-ostree distribution-gpg-keys jq systemd dhcp-client bind-export-libs iproute selinux-policy selinux-policy-targeted policycoreutils python3 python3-gobject-base buildah podman skopeo iptables iptables-libs cryptsetup gdisk xfsprogs e2fsprogs dosfstools shim grub2 powerpc-utils
+ find /usr/lib/coreos-assembler/ -type f
+ cat
++ id -u
++ cat /usr/lib/coreos-assembler/supermin-init-prelude.sh
+ chmod a+x /srv/tmp/supermin.prepare/init
+ cd /srv/tmp/supermin.prepare
+ tar -czf init.tar.gz --remove-files init
+ supermin --build /srv/tmp/supermin.prepare --size 5G -f ext2 -o /srv/tmp/supermin.build
+ echo /usr/lib/coreos-assembler/create_disk.sh --disk /dev/vda --buildid 31.20200110.dev.0 --imgid fedora-coreos-31.20200110.dev.0-qemu.ppc64le.qcow2 --grub-script /usr/lib/coreos-assembler/grub.cfg --kargs '"mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=hvc0,115200n8 ignition.platform.id=qemu"' --osname fedora-coreos --ostree-ref fedora/ppc64le/coreos/testing-devel --ostree-remote fedora --ostree-repo /srv/tmp/repo --save-var-subdirs NONE --rootfs-size 2336M --rootfs xfs --boot-verity
+ cachedisk=()
+ '[' -f /srv/cache/cache.qcow2 ']'
+ srcvirtfs=()
+ '[' -L /srv/src/config ']'
+ scsibus=bus=pci.0,addr=0x3
+ arch_args=
+ case $arch in
+ qemu-system-ppc64 -accel kvm -smp 16 -vga none -nodefaults -nographic -m 2048 -no-reboot -cpu host -kernel /srv/tmp/supermin.build/kernel -initrd /srv/tmp/supermin.build/initrd -netdev user,id=eth0,hostname=supermin -device virtio-net-pci,netdev=eth0 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -drive if=none,id=drive-scsi0-0-0-0,snapshot=on,file=/srv/tmp/supermin.build/root -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -virtfs local,id=workdir,path=/srv,security_model=none,mount_tag=workdir -serial stdio -append 'root=/dev/sda console=hvc0 selinux=1 enforcing=0 autorelabel=1' -drive if=virtio,id=target,format=qcow2,file=/srv/tmp/build/fedora-coreos-31.20200110.dev.0-qemu.ppc64le.qcow2.tmp,cache=unsafe

SLOF **********************************************************************
QEMU Starting
 Build Date = Jul 24 2019 00:00:00
 FW Version = mockbuild@ release 20190114
 Press "s" to enter Open Firmware.

Populating /vdevice methods
Populating /vdevice/vty@71000000
Populating /vdevice/nvram@71000001
Populating /pci@800000020000000
                     00 0000 (D) : 1af4 1000    virtio [ net ]
                     00 0800 (D) : 1af4 1005    unknown-legacy-device*
                     00 1000 (D) : 1af4 1009    virtio [ network ]
Populating /pci@800000020000000/virtio-9p@2
                     00 1800 (D) : 1af4 1004    virtio [ scsi ]
Populating /pci@800000020000000/scsi@3
       SCSI: Looking for devices
          100000000000000 DISK     : "QEMU     QEMU HARDDISK    2.5+"
                     00 2000 (D) : 1af4 1001    virtio [ block ]
No NVRAM common partition, re-initializing...
Scanning USB 
Using default console: /vdevice/vty@71000000
Detected RAM kernel at 400000 (1e54e58 bytes) 

  Welcome to Open Firmware

  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Booting from memory...
OF stdout device is: /vdevice/vty@71000000
Preparing to boot Linux version 5.4.8-200.fc31.ppc64le (mockbuild@buildvm-ppc64le-05.ppc.fedoraproject.org) (gcc version 9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)) #1 SMP Mon Jan 6 16:29:22 UTC 2020
Detected machine type: 0000000000000101
command line: root=/dev/sda console=hvc0 selinux=1 enforcing=0 autorelabel=1
Max number of cores passed to firmware: 2048 (NR_CPUS = 2048)
Calling ibm,client-architecture-support...+ '[' '!' -f /srv/tmp/rc ']'
+ fatal 'Couldn'\''t find rc file, something went terribly wrong!'
+ echo 'fatal: Couldn'\''t find rc file, something went terribly wrong!'
fatal: Couldn't find rc file, something went terribly wrong!
+ exit 1
+ rc=1
+ set +x

Reproduction Steps

Build Fedora CoreOS on power9 bare matal machine.

Other Information

Fedora tracking BZ https://bugzilla.redhat.com/show_bug.cgi?id=1790436

[jcajka]

CC'ing @gustavold

[cgwalters]

What commit of cosa is this? We just landed https://github.com/coreos/coreos-assembler/pull/1038 which might be related.

[jcajka]

This is on current master(P8 bare metal works). Also for record, this happens with virt-preview repo enabled(https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/). More precisely also with virt-preview repo enabled(not only with f31).

[cgwalters]

Why do you have virt-preview enabled? This seems likely to be a generic qemu issue. Does that qemu work at all? Does it work in a container?

[jcajka]

I have been working around and investigating other virt related issues. This seems to be cause by some issue in SLOF. With most recent SLOF this seems to be fixed, updated BZ.

[cgwalters]

Sounds like this was a bug in the virt stack; closing, but feel free to reopen if there's something to fix here!