Add ability to produce containers without zypper/rpm in them

[Vogtinator]

It should be possible to create container images without any package manager inside. Currently that's not easily possible without listing all packages manually in the delete section.

[davidcassany]

Have you tried uninstall instead of delete? I believe this should be what are you looking for.

Consider something like

<packages type="uninstall">
  <package name="zypper"/>
</package>

This will produce a call of chroot root_dir zypper rm -u --force-resolution zypper after running config.sh. Isn't it sufficient?

[Vogtinator]

That's what I tried, but it does not work:

[   77s] [ DEBUG   ]: 13:10:52 | system: ( 1/82) Removing zypper-1.14.19-1.1.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: ( 2/82) Removing procps-3.3.15-1.2.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: ( 3/82) Removing libzypp-17.10.3-1.1.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: ( 4/82) Removing libaugeas0-1.11.0-1.1.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: ( 5/82) Removing libprocps7-3.3.15-1.2.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: ( 6/82) Removing libsolv-tools-0.7.2-1.1.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: ( 7/82) Removing libproxy1-0.4.15-5.1.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: ( 8/82) Removing libgpgme11-1.12.0-2.1.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: ( 9/82) Removing libcurl4-7.63.0-1.2.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: (10/82) Removing libboost_thread1_68_0-1.68.0-3.2.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: (11/82) Removing libsystemd0-239-3.1.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: (12/82) Removing rpm-config-SUSE-0.g8-1.1.noarch [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: (13/82) Removing libzstd1-1.3.8-1.1.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: (14/82) Removing libxml2-2-2.9.8-2.3.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: (15/82) Removing libmodman1-2.0.1-18.2.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: (16/82) Removing gpg2-2.2.12-1.1.x86_64 [.....done]
[   77s] [ DEBUG   ]: 13:10:52 | system: Additional rpm output:
[   77s] [ DEBUG   ]: 13:10:52 | system: install-info: No such file or directory for /usr/share/info/gnupg.info.gz
[   78s] [ DEBUG   ]: 13:10:52 | system: (17/82) Removing libssh4-0.8.6-1.1.x86_64 [.....done]
[   78s] [ DEBUG   ]: 13:10:52 | system: (18/82) Removing libpsl5-0.20.2-1.3.x86_64 [.....done]
[   78s] [ DEBUG   ]: 13:10:52 | system: (19/82) Removing libnghttp2-14-1.34.0-2.1.x86_64 [.....done]
[   78s] [ DEBUG   ]: 13:10:52 | system: (20/82) Removing libboost_system1_68_0-1.68.0-3.2.x86_64 [.....done]
[   78s] [ DEBUG   ]: 13:10:52 | system: (21/82) Removing liblz4-1-1.8.3-1.1.x86_64 [.....done]
[   78s] [ DEBUG   ]: 13:10:52 | system: (22/82) Removing rpm-4.14.2.1-2.1.x86_64 [.....done]
[   78s] [ DEBUG   ]: 13:10:52 | system: Additional rpm output:
[   78s] [ DEBUG   ]: 13:10:52 | system: error: rpmdb: Packages: unable to flush: No such file or directory
[   78s] [ DEBUG   ]: 13:10:52 | system: error: db4 error(2) from dbenv->close: No such file or directory
[   78s] [ DEBUG   ]: 13:10:52 | system: (23/82) Removing pinentry-1.1.0-2.2.x86_64 [....error]
[   78s] [ DEBUG   ]: 13:10:52 | system: Abort, retry, ignore? [a/r/i] (a): a
[   78s] [ INFO    ]: Processing: [########################################] 100%
[   78s] [ ERROR   ]: 13:10:52 | KiwiPackagesDeletePhaseFailed: KiwiSystemDeletePackagesFailed: Package deletion failed: Removal of (80)pinentry-1.1.0-2.2.x86_64(@System) failed:
[   78s] Error: Subprocess failed. Error: RPM failed: Can't exec 'rpm' (No such file or directory).

[davidcassany]

Thats interesting, is there any OBS project I can look at with this failure? at first sight looks like there is some issue with how zypper resolves the dependencies, it is not removing RPM binary as the last one.

[Vogtinator]

Thats interesting, is there any OBS project I can look at with this failure? at first sight looks like there is some issue with how zypper resolves the dependencies, it is not removing RPM binary as the last one.

That will never work though. rpm itself has a huge list of requires, so all of those would need to be removed in one go, which zypper is unable to do.

[davidcassany]

Yes that makes sense... then it is hard to come with a solution for it. I'll think about it, I don't like the idea of changing how KIWI bootstraps to avoid installing zypper.

[Vogtinator]

AFAICT it wouldn't need many changes - just not doing any rpm or zypper calls in the chroot if not installed would work. For instance, it does some rpm -E %_rpmdb (or similiar) calls while it wouldn't be required.

Then only packages in the bootstrap sections could be installed, but that's actually quite matching.

[davidcassany]

Just verified that our bootstrap code is smart enough not to install the zypper package unless there some system packages listed. Meaning zypper will not get installed if you include only bootstrap packages in your image definition, without any <packages type="image"> section.

However I found an issue I'll try to fix, in our code there is a method that adapts the rpmdb initiated during bootstrap (using host rpm binary) to the expected format for the rpm binary present in the built image, our code here expects the rpm binary to be present into the image.

So for now you can build the image without zypper, you just need to make sure you add rpm package into the bootstrap (in my test is was pulled by zypper, so removing zypper triggered this error).

I did a tumbleweed build with the following list of packages:

<packages type="bootstrap">
    <package name="aaa_base"/>
    <package name="cracklib-dict-small"/>
    <package name="filesystem"/>
    <package name="openSUSE-release"/>
    <package name="shadow"/>
    <package name="ca-certificates"/>
    <package name="ca-certificates-mozilla"/>
    <package name="coreutils"/>
    <package name="iputils"/>
    <package name="openSUSE-build-key"/>
    <package name="krb5"/>
    <package name="netcfg"/>
    <package name="kubic-locale-archive"/>
    <package name="rpm"/>
  </packages>

[davidcassany]

Then only packages in the bootstrap sections could be installed, but that's actually quite matching.

Correct, but there is no other way to solve this issue. Either we install zypper and remove it afterwards (which seams quite complicated) or we only install in bootstrap phase using host package manager only.

I did a very quick and wild test trying to uninstall packages without chrooting and using hosts zypper with --root flag but I also got some issues when unstalling, thus I don't think this is a good solution.

[Vogtinator]

Looks good so far. I had a look why uninstalling does not work and it seems to be caused by the uninstallation of rpm confusing zypper. Not a bug in kiwi though.

[davidcassany]

Looks good so far. I had a look why uninstalling does not work and it seems to be caused by the uninstallation of rpm confusing zypper. Not a bug in kiwi though.

JFYI in case you want to further dig about that. You can avoid removing rpm on zypper uninstall if you request rpm installation in your packages section. This way rpm won't be marked as being installed as a dependency. I tried that, rpm is not removed, but zypper still fails due to some libzypp or probably libsolv issue. I fear it could be an issue related with rpmdb path, IIRC the issue I got was rpm complaining about no being capable to write the rpmdb. Unfortunately we still have some tools relaying on having the db in /var/lib/rpm like libsolve and probably others. So during the uninstall probably some configs and tricks around the rpmdb path are breaking. Just a mere guess.

[Vogtinator]

So during the uninstall probably some configs and tricks around the rpmdb path are breaking. Just a mere guess.

Unlikely, as rpm --root $root -qa still works, while zypper --root $root se -i does not.

[Vogtinator]

FYI, I reported the zypper bug as https://bugzilla.opensuse.org/show_bug.cgi?id=1122471

[Vogtinator]

Uninstalling packages doesn't work if rpm is not in the chroot:

[  114s] [ INFO    ]: 09:40:10 | Uninstall system packages (chroot)
[  114s] [ INFO    ]: 09:40:10 | --> package: bash
[  114s] [ INFO    ]: 09:40:10 | --> package: coreutils
[  114s] [ DEBUG   ]: 09:40:10 | EXEC: [chroot /usr/src/packages/KIWI-docker/build/image-root rpm -q bash]
[  114s] [ DEBUG   ]: 09:40:10 | EXEC: Failed with stderr: chroot: failed to run command ‘rpm’: No such file or directory
[  114s] , stdout: (no output on stdout)
[  114s] [ DEBUG   ]: 09:40:10 | EXEC: [chroot /usr/src/packages/KIWI-docker/build/image-root rpm -q coreutils]
[  114s] [ DEBUG   ]: 09:40:10 | EXEC: Failed with stderr: chroot: failed to run command ‘rpm’: No such file or directory
[  114s] , stdout: (no output on stdout)

[davidcassany]

Yes that's the expected behavior, We absolutely discard uninstalling anything in a not chrooted env. At least with chroot we are certain we do not face rpm or zypper incompatibilities between host and image. I guess the closed you can get to something like that is including rpm binary into the image and then cherry pick whatever you'd like to delete in type="delete" packages list or manually delete whatever you'd like in config.sh.

[Vogtinator]

We absolutely discard uninstalling anything in a not chrooted env.

Why? If something was installed with the host zypper, it can be uninstalled with the host zypper as well.

including rpm binary into the image and then cherry pick whatever you'd like to delete in type="delete"

That won't work out, adding rpm just to uninstall packages needs to be avoided and having to maintain a manual list of packages (including library sonames) really does not scale.

or manually delete whatever you'd like in config.sh.

That sounds like a nightmare, it would require deleting bash inside a bash script, removing rm with rm itself and so on...

[davidcassany]

We absolutely discard uninstalling anything in a not chrooted env.

Why? If something was installed with the host zypper, it can be uninstalled with the host zypper as well.

We support building images (system images and containers) from hosts up to two distros newer than the image being build (e.g. build Leap 42.3 from a Leap 15 host). We already have some code to handle issues created by bootstrapping with a different rpm version of what it is included into the image. I don't think we should change that behavior as it will lead to more issues and corner cases, jumping from host to image package manager should be avoided as much as possible, specially if this performs changes over the image itself.

More over KIWI is designed to build full systems at first instance and has a common prepare step where the root tree is being populated regardless of building containers or any other kind of image. Rethinking this procedure is something we are not considering.

including rpm binary into the image and then cherry pick whatever you'd like to delete in type="delete"

That won't work out, adding rpm just to uninstall packages needs to be avoided and having to maintain a manual list of packages (including library sonames) really does not scale.

I agree it does not scale, the question then is why they were installed at first instance. Do not include them into the bootstrap phase, this works as the result of (#917). Limiting to the very minimum what is included in to a container should be done at packaging level IMHO.

or manually delete whatever you'd like in config.sh.

That sounds like a nightmare, it would require deleting bash inside a bash script, removing rm with rm itself and so on...

Yes, so the solution is not adding them if you don't need them.

[Vogtinator]

jumping from host to image package manager should be avoided as much as possible, specially if this performs changes over the image itself.

Of course, but in this case there is no image package manager at all. In most cases the host and target are the same distro as well.

More over KIWI is designed to build full systems at first instance and has a common prepare step where the root tree is being populated regardless of building containers or any other kind of image. Rethinking this procedure is something we are not considering.

There shouldn't be any rethinking required - only some commands need to be adjusted slightly.

I agree it does not scale, the question then is why they were installed at first instance. Do not include them into the bootstrap phase, this works as the result of (#917). Limiting to the very minimum what is included in to a container should be done at packaging level IMHO.

Yes, so the solution is not adding them if you don't need them.

In this case to run a config.sh script to create a symlink. That was unavoidable as the container runtime required a file /pause while the RPM installs /usr/bin/kubic-pause. Doing that in the RPM itself would not be allowed in the distro.

[Vogtinator]

Ping - can this be reopened? It's still impossible to uninstall packages if zypper is not part of the image.

[Vogtinator]

Ping again.

[Vogtinator]

Ping again. Can this at least be reopened?

[Vogtinator]

A month later: Ping!

[davidcassany]

So sorry I missed it... Reopening so it does not get buried under other stuff

Regarding the issue I still struggle to find a reasonable use case for that. I have the feeling the cases that are not currently covered by KIWI are just a couple of cornercases:

1. Using <packages type="uninstall"> is not possible if zypper is not part of the image. However you can still use <packages type="delete">
2. Use <packages type="delete"> to remove the packager manager stack fails

Anyway I see some options:

1. On <packages type="install">` fallback to host if the package manager is not found.
2. Add a new type to uninstall packages from host. <packages type="deinstall">
3. Change behavior for type=uninstall to use always the host package manager

Does anyone have further ideas? I am not happy with any of the above options. In fact I'd say the only clean implementation of the above are 2nd and 3rd. For the 3rd this is a behavior change that is not backward compatible, thus I don't we should do so. Also using host's package manager to uninstall packages is dangerous.

For the 2nd option I think the implementation would be clean and simple, however I really dislike the fact of adding yet another way to delete packages.

For the 1st I believe this is a misleading behavior.

Reading back to the issue history it seams to be that the specific motivation for this issue is that the pause image for kubic requires a symlink (from /pause to /usr/bin/pause). Isn't it enough to set the entrypoint to /usr/bin/pause? why /pause is needed? Also, there are alternatives, why aren't you adding the symlink into the root overlay tree? or as archive tarball in packages list? This way no code needs to be executed in config.sh and the package does not require to set the /pause symlink.

[Vogtinator]

So sorry I missed it... Reopening so it does not get buried under other stuff

Thanks!

Regarding the issue I still struggle to find a reasonable use case for that. I have the feeling the cases that are not currently covered by KIWI are just a couple of cornercases:

We have an increasing number of images without zypper (or even rpm) inside, so regularly new use cases appear.

There's also the case of Requires(pre,post) dependencies. Those get installed as part of the initial setup, but are no longer required after the transaction finished. This is for example shadow in the base container, which is used to create the user accounts. Though that's arguably something that could be done at a higher level by kiwi again.

1. Using <packages type="uninstall"> is not possible if zypper is not part of the image. However you can still use <packages type="delete">
2. Use <packages type="delete"> to remove the packager manager stack fails

Anyway I see some options:

1. On <packages type="install">` fallback to host if the package manager is not found.

We have <packages type="bootstrap"> for that. Did you mean <packages type="uninstall">`?

2. Add a new type to uninstall packages from host. <packages type="deinstall">

Or even more generically: <packages type="..." chrooted="no"> - would also apply to bootstrap/install.

3. Change behavior for type=uninstall to use always the host package manager

Does anyone have further ideas? I am not happy with any of the above options. In fact I'd say the only clean implementation of the above are 2nd and 3rd. For the 3rd this is a behavior change that is not backward compatible, thus I don't we should do so. Also using host's package manager to uninstall packages is dangerous.

Yep, if it can be avoided.

For the 2nd option I think the implementation would be clean and simple, however I really dislike the fact of adding yet another way to delete packages.

For the 1st I believe this is a misleading behavior.

Reading back to the issue history it seams to be that the specific motivation for this issue is that the pause image for kubic requires a symlink (from /pause to /usr/bin/pause). Isn't it enough to set the entrypoint to /usr/bin/pause? why /pause is needed?

That was a bug in kubernetes. Meanwhile it does not hardcode /pause anymore.

Also, there are alternatives, why aren't you adding the symlink into the root overlay tree? or as archive tarball in packages list? This way no code needs to be executed in config.sh and the package does not require to set the /pause symlink.

IIRC that's what I ended up doing as a workaround.

[davidcassany]

We have discussed it internally once again and we decided to close the issue, as we still do not see strong use cases for that.

1. Deleting RPM stack in derived images is a no sense
2. Building base images without RPM stack is already possible (only bootstrap packages)
3. Building derived images without RPM stack is also possible (only bootstrap packages on top of a base without RPM)
4. Deleting packages right after the root-tree installation and configuration is possible with some limitations:
   • The removal is done from the root-tree (thus removing the RPM stack itself might be tricky)
   • Requires zypper or at least rpm to be installed into the image (zypper or direct rpm calls are used to remove packages)
5. Images that do not include RPM stack, shell and other OS fundamental tools are still possible to be created as KIWI supports filling the root-tree from a given tarball

Use cases that are not covered by these options are minimal. KIWI pretends to be generic appliance builder and we believe we already provide a wide range of customization possibilities and appliances types. We do not want to provide new mechanisms to remove packages that could be misused or include undesired side effects.

[Vogtinator]

Images that do not include RPM stack, shell and other OS fundamental tools are still possible to be created as KIWI supports filling the root-tree from a given tarball

KIWI is the tool we have to create such a tarball though.

Use cases that are not covered by these options are minimal. KIWI pretends to be generic appliance builder and we believe we already provide a wide range of customization possibilities and appliances types. We do not want to provide new mechanisms to remove packages that could be misused or include undesired side effects.

Now that we have multiple "BCI" containers of which many flavors do not include rpm and/or zypper, this issue is resurfacing again. In some cases, packages are installed, then used in config.sh for generating some necessary files and later deleted again.

Unless the container image contains the zypper stack, type="uninstall" fails. There's no way to cleanly uninstall any package which was previously installed via type="bootstrap".

e.g. this simple example is broken:

<packages type="uninstall">
<package name="foobar"/>
</packages>
<packages type="bootstrap">
<package name="foobar"/>
</packages>

Using delete instead of uninstall works, but does not clean up dependencies. Thus libraries and other packages are left behind.