livemedia-creator with "--make-pxe-live" produces error "non-existent device" for Fedora 25/26

[ruzickap]

Hello.

I have a Fedora 25 using "lorax-25.19-1.fc25.x86_64" package to build live pxe boot squashfs image.

I'm using the following command

livemedia-creator --make-pxe-live --image-name=my_fedora_img --qcow2 --tmp=/var/tmp/a --ks fedora25-my.ks --iso=Fedora-Workstation-netinst-x86_64-25-1.3.iso

The "fedora25-my.ks" is pretty basic:

url --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch
lang en_US.UTF-8
keyboard us
timezone --utc Etc/UTC
network --bootproto=dhcp --device=link --activate
rootpw root
shutdown
zerombr
clearpart --all --initlabel --disklabel=msdos
part / --size 5120 --fstype ext4
bootloader --timeout=1 --append="no_timer_check console=tty1 console=ttyS0,115200n8"

%packages
@core
dracut-live
%end

Qemu did the job and generates the image, but then I got the error:

Running... qemu-img create -f qcow2 /var/tmp/a/my_fedora_img 5370806272
Formatting '/var/tmp/a/my_fedora_img', fmt=qcow2 size=5370806272 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
Return code: 0
Running... qemu-system-x86_64 -nodefconfig -m 8192 -smp 2 --machine accel=kvm -kernel /var/tmp/a/lorax.imgutils.zxwe7haj/isolinux/vmlinuz -initrd /var/tmp/a/lmc-initrd-xhshv9qs.img -drive file=/var/tmp/a/my_fedora_img,cache=unsafe,discard=unmap,format=qcow
2 -drive file=/home/ruzickap/data2/iso/Fedora-Workstation-netinst-x86_64-25-1.3.iso,media=cdrom,readonly=on -append ks=file:/fedora25-my.ks inst.stage2=hd:LABEL=Fedora-WS-dvd-x86_64-25 inst.text inst.cmdline -nographic -display vnc=127.0.0.1:0 -device virt
io-serial-pci,id=virtio-serial0 -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.fedoraproject.anaconda.log.0 -chardev socket,id=charchannel0,host=127.0.0.1,port=59921 -object rng-random,id=virtio-rng0,filename=/de
v/random -device virtio-rng-pci,rng=virtio-rng0,id=rng0,bus=pci.0,addr=0x9
Return code: 0
Running... umount /var/tmp/a/lorax.imgutils.zxwe7haj
Return code: 0
Running... kpartx -v -a -s /var/tmp/a/my_fedora_img
Return code: 0
Running... kpartx -d -s /var/tmp/a/my_fedora_img
Return code: 0
Running... /usr/sbin/fsck.ext4 -y -f -E discard /var/tmp/a/lmc-work-klz2ladw/squashfs_root/LiveOS/rootfs.img
e2fsck 1.43.3 (04-Sep-2016)
/usr/sbin/fsck.ext4: No such file or directory while trying to open /var/tmp/a/lmc-work-klz2ladw/squashfs_root/LiveOS/rootfs.img
Possibly non-existent device?
Return code: 8

Do you have any idea why I'm getting it or what may be wrong? (I do have the same error with Fedora 26 Beta)

Thank you

PetrR

[bcl]

Try it without --qcow2 qcow2 only makes sense in some cases, --make-disk for example. It cannot be mounted by the host so it isn't possible to make any further changes to the image once it has been installed in the qemu session. I should add a check for this situation to make it clearer when you can and cannot use qcow2.

[ruzickap]

Thank you.

Without the --qcow2 parameter I can move "further", but it ends with error:

2017-06-29 18:30:45,616: dracut args = ['--xz', '--add', 'livenet dmsquash-live convertfs pollcdrom qemu qemu-net', '--omit', 'plymouth', '--no-hostonly', '--debug', '--no-early-microcode']
2017-06-29 18:30:45,630: rebuilding initramfs-4.8.6-300.fc25.x86_64.img
2017-06-29 18:30:46,088: failed to unmount /var/tmp/a/lorax.imgutils.ebqfx61_. retrying (1/3)...
2017-06-29 18:30:47,186: failed to unmount /var/tmp/a/lorax.imgutils.ebqfx61_. retrying (2/3)...
Traceback (most recent call last):
  File "/usr/sbin/livemedia-creator", line 1167, in make_live_images
    rebuild_initrds_for_live(opts, joinpaths(mnt_dir, sys_root), work_dir)
  File "/usr/sbin/livemedia-creator", line 507, in rebuild_initrds_for_live
    runcmd(cmd, root=sys_root_dir)
  File "/usr/lib/python3.5/site-packages/pylorax/executils.py", line 341, in runcmd
    return execWithRedirect(cmd[0], cmd[1:], **kwargs)
  File "/usr/lib/python3.5/site-packages/pylorax/executils.py", line 228, in execWithRedirect
    env_add=env_add, reset_handlers=reset_handlers, reset_lang=reset_lang)[0]
  File "/usr/lib/python3.5/site-packages/pylorax/executils.py", line 201, in _run_program
    raise subprocess.CalledProcessError(proc.returncode, argv, output)
subprocess.CalledProcessError: Command '['dracut', '--nomdadmconf', '--nolvmconf', '--xz', '--add', 'livenet dmsquash-live convertfs pollcdrom qemu qemu-net', '--omit', 'plymouth', '--no-hostonly', '--debug', '--no-early-microcode', '/results/initramfs-4.8.6-300.fc25.x86_64.img', '4.8.6-300.fc25.x86_64']' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/livemedia-creator", line 1476, in <module>
    main()
  File "/usr/sbin/livemedia-creator", line 1454, in main
    result_dir = make_live_images(opts, work_dir, disk_img)
  File "/usr/sbin/livemedia-creator", line 1169, in make_live_images
    umount(joinpaths(mnt_dir, sys_root, "boot"), delete=False)
  File "/usr/lib/python3.5/site-packages/pylorax/imgutils.py", line 318, in __exit__
    umount(self.mnt)
  File "/usr/lib/python3.5/site-packages/pylorax/imgutils.py", line 211, in umount
    rv = runcmd(cmd)
  File "/usr/lib/python3.5/site-packages/pylorax/executils.py", line 341, in runcmd
    return execWithRedirect(cmd[0], cmd[1:], **kwargs)
  File "/usr/lib/python3.5/site-packages/pylorax/executils.py", line 228, in execWithRedirect
    env_add=env_add, reset_handlers=reset_handlers, reset_lang=reset_lang)[0]
  File "/usr/lib/python3.5/site-packages/pylorax/executils.py", line 201, in _run_program
    raise subprocess.CalledProcessError(proc.returncode, argv, output)
subprocess.CalledProcessError: Command '['umount', '/var/tmp/a/lorax.imgutils.ebqfx61_']' returned non-zero exit status 32

I can see the mounted filesystem:

# mount | grep '/var/tmp/a/lorax.imgutils.ebqfx61_'
/var/tmp/a/lmc-work-4fmobxf3/squashfs_root/LiveOS/rootfs.img on /var/tmp/a/lorax.imgutils.ebqfx61_ type ext4 (rw,relatime,seclabel,data=ordered)
/dev/mapper/fedora-root on /var/tmp/a/lorax.imgutils.ebqfx61_/results type ext4 (rw,relatime,seclabel,data=ordered)
/dev/mapper/fedora-root on /var/tmp/a/lorax.imgutils.ebqfx61_/var/tmp type ext4 (rw,relatime,seclabel,data=ordered)

It looks like the script is trying to unmount /var/tmp/a/lorax.imgutils.ebqfx61_ without unmounting the /var/tmp/a/lorax.imgutils.ebqfx61_/results and /var/tmp/a/lorax.imgutils.ebqfx61_/var/tmp first.

Here is the proper "unmount" sequence:

# umount /var/tmp/a/lorax.imgutils.ebqfx61_
umount: /var/tmp/a/lorax.imgutils.ebqfx61_: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
# umount /var/tmp/a/lorax.imgutils.ebqfx61_/results
# umount /var/tmp/a/lorax.imgutils.ebqfx61_/var/tmp
# umount /var/tmp/a/lorax.imgutils.ebqfx61_
#

Any idea how I can get over it? Is it a bug or do I have something misconfigured?

[bcl]

I have no idea why something would be mounted on /results or /var/tmp inside the rootfs. Check the logs (especially from anaconda) for anything relating to those paths.

[ruzickap]

In the program.log I can see:

Running... mount -o loop /var/tmp/a/lmc-work-qb7g_g4m/squashfs_root/LiveOS/rootfs.img /var/tmp/a/lorax.imgutils.4y6h5lss
Return code: 0
Running... mount -o bind /var/tmp/a/lorax.imgutils.4y6h5lss/boot /var/tmp/a/lorax.imgutils.4y6h5lss//boot
Return code: 0
Running... mount -o bind /var/tmp/a/lmc-work-qb7g_g4m /var/tmp/a/lorax.imgutils.4y6h5lss//results
Return code: 0
Running... mount -o bind /var/tmp /var/tmp/a/lorax.imgutils.4y6h5lss//var/tmp
Return code: 0
Running... dracut --nomdadmconf --nolvmconf --xz --add livenet dmsquash-live convertfs pollcdrom qemu qemu-net --omit plymouth --no-hostonly --debug --no-early-microcode /results/initramfs-4.8.6-30
0.fc25.x86_64.img 4.8.6-300.fc25.x86_64

and then it's not properly "unmounted".

Full log attached: logs.zip

[nicholseric]

I saw some weird issues with mounting\unmounting when using a virtual esx guest and passing certain disks through to the guest. A reboot of the guest and then running anaconda-cleanup before running livemedia-creator again seemed to fix it.

[bcl]

Oh, ha, I wrote that and totally forgot about it. I think the unmount problem is fallout from a dracut failure since there's a CalledProcessError there. Is there anything else in the logs that looks like a dracut error? Also, it's probably time to move this to bugzilla.redhat.com and file a bug against lorax with all the logs attached.

[ruzickap]

I created the Bug 1467506 then. All the details are there (including logs + kickstart + example command).

[ruzickap]

It looks like package dracut-live is mandatory for successfully build. Once I add dracut-live to the %packages section it started working...