KIWI on OBS stops with an error when building a Fedora 28 based image

[mmohring]

Problem description

When creating an image with KIWI in OBS for Fedora 28, KIWI ends up in an error. With Fedora 27 it worked (still works).

Expected behaviour

The Image Creation for Fedora 28 should work als for all Fedora versions before.

Steps to reproduce the behaviour

1. Use an image description for Fedora 27
2. Build the python-kiwi package for Fedora 28
3. Modify image description to build a Fedora 28 Image
4. The process ends up in an error, see logfile attached

OS and Software information

• KIWI version: 9.15.3
• Operating system: Fedora 28 target, in OBS also host
• OBS version: 2.9 / 2.10

This is the kiwi config file producing a Fedora 28 Image: config.txt

This is the logfile for the image build: logfile.txt

[schaefi]

Hmm, the log said:

[   87s] Can not read configuration: //tmpnvkf_2lp , ignoring
[   87s] Error: Failed to synchronize cache for repo 'updates'

I wonder why the temporary config file has been gone, it should still be there, but this is not a problem for dnf, as it ignores it. However it then can't synchronize the cache, maybe an addition makecache call is required

Just guessing in the wild, I'll do some tests

[mmohring]

Hi @schaefi, I had been playing with Clear Linux inside OBS. I could create a working OBS projconf and patched build a bit to get it working. After adaption of python-kiwi.spec, I could package it. Long story short outcome: Clear Linux uses the same version of DNF, and after getting kiwi working on it, it runs into the same DNF error. As mentioned, Fedora 25 - 27 works fine.

[schaefi]

I have moved our test build to fc28, see here:

• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86/test-image-fedora-vmx

Currently in building state...

[schaefi]

As the same code based builds successfully with dnf in fc25 - 27 I guess a change in dnf causes the problem. Maybe @Conan-Kudo can help us

[Conan-Kudo]

For Fedora 27 onward, the starfield theme doesn't exist.

Install grub2-breeze-theme, change bootloader theme to breeze.

The filesystem should be ext4.

[schaefi]

@Conan-Kudo I've added that, thanks

I guess it will be unrelated to the dnf error we see. Once I have a failed build at hand I can tell you more. Maybe the logs attached by @mmohring triggers an idea

[Conan-Kudo]

@mmohring Also, could you please try the kiwi package in Fedora 28 to build for Fedora 28?

[mmohring]

Hi @Conan-Kudo, I had tested the issue as requested with Fedora 28 supplied kiwi-cli-9.14.0-1.fc28 and its dependencies. It resulted in the same error.

[Conan-Kudo]

@mmohring In or out of OBS?

[mmohring]

In OBS.

[Conan-Kudo]

Can you try a variant of your kiwi config on a Fedora 28 system? That will help to rule out whether or not OBS is misfiring in some manner.

[mmohring]

Just tell me how I should modify the config.kiwi and I try it. Just give me a modified config.kiwi. Do you need a log using kiwi-cli-9.14.0-1.fc28 ? Its pretty easy to use kiwi from Fedora inside OBS, so you can test it also inside b.o.o.

[Conan-Kudo]

Try this on kiwi on a Fedora machine (not in OBS): config.txt

[schaefi]

I also can reproduce the problem with the build in Virtualization:Appliances:Images:Testing_x86/test-image-fedora-vmx.

I tried a local build with "osc build" but that failed to setup the chroot env with:

[   11s] [129/485] installing filesystem-3.8-2.fc28
[   11s] error: unpacking of archive failed on file /lib64: cpio: File from package already exists as a directory in system

so a local osc build failed before kiwi was called. I have no other fc28 build machine at hand right now. Hmm

[Conan-Kudo]

@schaefi I think we have an ordering issue. filesystem should be the first package installed, which sets up the filesystem tree correctly.

[Conan-Kudo]

[  167s] [ ERROR   ]: 17:18:59 | KiwiInstallPhaseFailed: System package installation failed: Can not read configuration: //tmpbdutefqu , ignoring
[  167s] Can not read configuration: //tmpbdutefqu , ignoring
[  167s] Error: Failed to synchronize cache for repo 'updates'

This error is definitely caused by KIWI <-> OBS. There is no network access for the default repos in fedora-repos to work (which are being read because it can't use the generated config). The generated config is broken because //tmpbdutefqu doesn't exist.

[Conan-Kudo]

I can reproduce this with kiwi-9.15.1-1.fc28, too.

Note the difference between bootstrap and main phases:

Bootstrap phase:

DEBUG: EXEC: [bash -c dnf -c /root/build/samples/fedora/vmroot/fedora-28/image-root/tmpktcyh5qg -y --installroot /root/build/samples/fedora/vmroot/fedora-28/image-root --setopt=install_weak_deps=False ins
tall basesystem dnf filesystem grub2-efi-x64 grub2-efi-x64-modules shim-x64]
DEBUG: bootstrap: Last metadata expiration check: 0:00:01 ago on Tue Jun  5 05:22:30 2018.
DEBUG: bootstrap: Dependencies resolved.
DEBUG: bootstrap: ================================================================================
DEBUG: bootstrap:  Package                     Arch   Version                      Repository
DEBUG: bootstrap:                                                                            Size
DEBUG: bootstrap: ================================================================================
DEBUG: bootstrap: Installing:
DEBUG: bootstrap:  basesystem                  noarch 11-5.fc28                    Fedora28 9.4 k
DEBUG: bootstrap:  dnf                         noarch 2.7.5-12.fc28                Fedora28 352 k
DEBUG: bootstrap:  filesystem                  x86_64 3.8-2.fc28                   Fedora28 1.1 M

Main phase:

DEBUG: EXEC: [bash -c chroot /root/build/samples/fedora/vmroot/fedora-28/image-root dnf -c //tmpktcyh5qg -y --setopt=install_weak_deps=False install dhclient grub2 grub2-breeze-theme grubby kernel plymout
h-theme-charge rsyslog selinux-policy-targeted && chroot /root/build/samples/fedora/vmroot/fedora-28/image-root dnf -c //tmpktcyh5qg -y --setopt=install_weak_deps=False group install "network-server"]
DEBUG: system: Fedora 28 - x86_64 - Updates                     44 MB/s |  14 MB     00:00
DEBUG: system: Fedora 28 - x86_64                               63 MB/s |  60 MB     00:00
DEBUG: system: Last metadata expiration check: 0:00:06 ago on Tue Jun  5 09:23:30 2018.
DEBUG: system: Dependencies resolved.
DEBUG: system: ================================================================================
DEBUG: system:  Package                   Arch    Version                       Repository
DEBUG: system:                                                                            Size
DEBUG: system: ================================================================================
DEBUG: system: Installing:
DEBUG: system:  dhcp-client               x86_64  12:4.3.6-20.fc28              updates  306 k
DEBUG: system:  grub2-breeze-theme        x86_64  5.12.5-1.fc28                 updates  3.5 M
DEBUG: system:  grub2-pc                  x86_64  1:2.02-34.fc28                fedora    28 k

It actually downloaded the metadata a second time, using the official fedora repo definitions that are shipped in fedora-repos.

[schaefi]

I think you are right, I'm just wondering why this is not a generic issue. If for some reason the temporary config gets deleted before the chrooted dnf call it should be always failing, but it works with the same kiwi on e.g fc25, that's weird

For debugging I need a fc28 environment, do you have an idea how to fix the ordering issue such that osc build would work ? Thanks

[schaefi]

Nevermind, I'll try to reproduce this outside obs

[Conan-Kudo]

Further investigation indicates the file isn't gone. It appears that //foo doesn't resolve to /foo anymore.

[Conan-Kudo]

For me, my local builds always succeed because I'm not using custom repos (because kiwi is now in Fedora repositories, so it just works), but I can observe this behavior pretty easily, as noted earlier.

[schaefi]

Hmm, if I could only reproduce it.

A local build outside of obs just worked for me:

sudo kiwi-ng --type vmx system build --description ~ms/Project/kiwi-descriptions/fedora/x86_64/fedora-28.0-JeOS/ --target-dir /tmp/mytest/

[schaefi]

@Conan-Kudo I'm sure you are right and all is caused by the missing temporary config file, but I don't see what deleted it

[Conan-Kudo]

@schaefi The worst part is that it wasn't deleted. It's still there. The path isn't resolving.

That is, for DNF, somehow //foo no longer auto-resolves to /foo. It might be a Python change.

[schaefi]

Further investigation indicates the file isn't gone. It appears that //foo doesn't resolve to /foo anymore.

Oh, so -c //foo is different from -c /foo ?

[Conan-Kudo]

Yes. Apparently so.

@dmach, @mhatina, @j-mracek, do you guys have any idea why DNF would no longer auto-resolve //foo to /foo?

[mmohring]

The bug ( "//" not equivalent to "/" for dnf ) appears to be not limited to Fedora. Clear Linux uses dnf 2.7.5 and libdnf 0.11.1. Fedora 28 uses dnf 2.7.5 and libdnf 0.11.1, not checked the patchlevels or other dependent packages involved. Both stop with the same error.

[j-mracek]

Please @Conan-Kudo can you provide an example where argument worked including dnf version? I tried "dnf install //etc/dnf/dnf.conf" with dnf-2.7.3, 3.0.0, and with yum with no success.

[Conan-Kudo]

@j-mracek This was working in Fedora 25, per @schaefi's comment. And it worked in Fedora 27 for me, too: http://ftp1.nluug.nl/os/Linux/distr/opensuse/repositories/home:/Pharaoh_Atem:/procdump:/appliance/images/LimeJeOS-Fedora-27.0.x86_64-0.0.1-Build5.24.packages

[Conan-Kudo]

Using this configuration, which you can run using kiwi on Fedora 28 just fine: https://build.opensuse.org/package/show/home:Pharaoh_Atem:procdump:appliance/procdump_appliance.fc27

[Conan-Kudo]

@j-mracek And this is about the --config switch for arbitrary config files, not passing as an argument for requesting a package.