search

konflux-ci / rpm-lockfile-prototype

2 stars 9 forks source link

dealing with modularity #24

Closed brianwcook closed 4 days ago

brianwcook commented 1 month ago

Apologies for the long post, but I am not quite sure what information here is useful / not.

What, if anything, is needed in order to make the rpm-lockfile-prototype tool produce a useable lock file when there is a module in the list of rpms? We have hit an issue when trying to install clang using a lockfile produced by this tool & UBI8. The user is not intentionally trying to use modularity, but dnf installing clang in ubi8 will enable a default module on its own. The lockfile produced includes this entry (the yaml doesn't render quite right):

- url: https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/x86_64/appstream/os/Packages/c/clang-17.0.6- 1.module+el8.10.0+20808+e12784c0.x86_64.rpm
      repoid: ubi-8-appstream-rpms
      size: 86972
      checksum: sha256:69fce4b20704ca5e9ef5913693642e0b846d08da550e200c618cd2fb195be03d
      name: clang
      evr: 17.0.6-1.module+el8.10.0+20808+e12784c0
      sourcerpm: clang-17.0.6-1.module+el8.10.0+20808+e12784c0.src.rpm

Installing using this lockfile and a repo generated by cachi2 produces:

No available modular metadata for modular package 'clang-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'clang-libs-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'clang-resource-filesystem-17.0.6-1.module+el8.10.0+20808+e12784c0.noarch', it cannot be installed on the system
No available modular metadata for modular package 'compiler-rt-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'libomp-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'libomp-devel-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'llvm-libs-17.0.6-2.module+el8.10.0+21256+978ccea6.x86_64', it cannot be installed on the system
Error: No available modular metadata for modular package

I can't tell if the issue lies with the lockfile, or with cachi2. Must we have a modules.yaml file before calling createrepo to make this work, or can we 'flatten' the repo so that there is only one version of clang and still have working dependency resolution?

Manually adding the modules.yaml to into the generated RPM repo from ubi8 appstream repo got me to:

[root@530cffa19857 /]# dnf install clang
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

CustomRepository                                                                                                                                                                                                          75 MB/s |  88 kB     00:00    
Last metadata expiration check: 0:00:01 ago on Wed Jul 24 20:46:56 2024.
Error: 
 Problem: package clang-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64 from customrepo requires gcc-c++, but none of the providers can be installed
  - package gcc-c++-8.5.0-22.el8_10.x86_64 from customrepo requires gcc = 8.5.0-22.el8_10, but none of the providers can be installed
  - conflicting requests
  - nothing provides libgomp.so.1()(64bit) needed by gcc-8.5.0-22.el8_10.x86_64 from customrepo
  - nothing provides binutils >= 2.30-17 needed by gcc-8.5.0-22.el8_10.x86_64 from customrepo
  - nothing provides glibc-devel >= 2.2.90-12 needed by gcc-8.5.0-22.el8_10.x86_64 from customrepo
  - nothing provides libgomp = 8.5.0-22.el8_10 needed by gcc-8.5.0-22.el8_10.x86_64 from customrepo
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@530cffa19857 /]# dnf install gcc
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:00:33 ago on Wed Jul 24 20:46:56 2024.
Error: 
 Problem: conflicting requests
  - nothing provides libgomp.so.1()(64bit) needed by gcc-8.5.0-22.el8_10.x86_64 from customrepo
  - nothing provides glibc-devel >= 2.2.90-12 needed by gcc-8.5.0-22.el8_10.x86_64 from customrepo
  - nothing provides binutils >= 2.30-17 needed by gcc-8.5.0-22.el8_10.x86_64 from customrepo
  - nothing provides libgomp = 8.5.0-22.el8_10 needed by gcc-8.5.0-22.el8_10.x86_64 from customrepo
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@530cffa19857 /]#

Adding all those things back into the rpms.in file and trying to produce a lockfile then produces:


(venv) bcook@bcook-thinkpadp16vgen1:~/cachi2-modularity/buildroot$ rpm-lockfile-prototype -f Dockerfile rpms.in.yaml --outfile rpms.lock-new.yaml
INFO:root:Running solver for x86_64
INFO:root:$ skopeo --override-arch=amd64 copy docker://registry.access.redhat.com/ubi8/go-toolset:1.21 dir:/tmp/tmpppu_uprh
Getting image source signatures
Checking if image destination supports signatures
Copying blob be575238ea98 done   | 
Copying blob b6a366ac05e5 done   | 
Copying blob 7027f4e4058b done   | 
Copying blob 8694db102e5b done   | 
Copying config ac107e1047 done   | 
Writing manifest to image destination
Storing signatures
INFO:root:Extracting rpmdb from layer 8694db102e5bd27fa30106f87d5a0f0c5ccccac0e5cc38ba56080d7559377096
INFO:root:Extracting rpmdb from layer 7027f4e4058bde8aaa497e47562e962c293039ba16f5fbfd07ff43a0d1dbd5a2
INFO:root:Extracting rpmdb from layer be575238ea985ef824635fbeaf7b33eaed98ef6ba1db1822ac0714f509304d17
INFO:root:Extracting rpmdb from layer b6a366ac05e5458bcf8b40dd8802b77ffbbe792e074a976b85ea818a92607e05
INFO:dnf:Added ubi-8-baseos-rpms repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/baseos/os
INFO:dnf:Added ubi-8-baseos-debug-rpms repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/baseos/debug
INFO:dnf:Added ubi-8-baseos-source repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/baseos/source/SRPMS
INFO:dnf:Added ubi-8-appstream-rpms repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/appstream/os
INFO:dnf:Added ubi-8-appstream-debug-rpms repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/appstream/debug
INFO:dnf:Added ubi-8-appstream-source repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/appstream/source/SRPMS
INFO:dnf:Added ubi-8-codeready-builder-rpms repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/codeready-builder/os
INFO:dnf:Added ubi-8-codeready-builder repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/codeready-builder/os
INFO:dnf:Added ubi-8-codeready-builder-debug-rpms repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/codeready-builder/debug
INFO:dnf:Added ubi-8-codeready-builder-source repo from https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/$basearch/codeready-builder/source/SRPMS
Traceback (most recent call last):
  File "/home/bcook/.local/bin/rpm-lockfile-prototype", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/bcook/.local/lib/python3.12/site-packages/rpm_lockfile/__init__.py", line 433, in main
    process_arch(
  File "/home/bcook/.local/lib/python3.12/site-packages/rpm_lockfile/__init__.py", line 274, in process_arch
    packages, sources = resolver(
                        ^^^^^^^^^
  File "/home/bcook/.local/lib/python3.12/site-packages/rpm_lockfile/__init__.py", line 213, in resolver
    base.resolve(allow_erasing=allow_erasing)
  File "/usr/lib/python3.12/site-packages/dnf/base.py", line 934, in resolve
    raise exc
dnf.exceptions.DepsolveError: 
 Problem 1: conflicting requests
  - nothing provides /usr/libexec/platform-python needed by environment-modules-4.5.2-4.el8.x86_64 from ubi-8-baseos-rpms
 Problem 2: conflicting requests
  - nothing provides /usr/libexec/platform-python needed by python3-libsemanage-2.9-9.el8_6.x86_64 from ubi-8-baseos-rpms
 Problem 3: conflicting requests
  - nothing provides /usr/libexec/platform-python needed by policycoreutils-python-utils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms
 Problem 4: package cmake-3.26.5-2.el8.x86_64 from ubi-8-appstream-rpms requires cmake-rpm-macros = 3.26.5-2.el8, but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by cmake-rpm-macros-3.26.5-2.el8.noarch from ubi-8-appstream-rpms
 Problem 5: package python3-policycoreutils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms requires python3-libsemanage >= 2.9-7, but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by python3-libsemanage-2.9-9.el8_6.x86_64 from ubi-8-baseos-rpms
 Problem 6: package python3-setools-4.3.0-5.el8.x86_64 from ubi-8-baseos-rpms requires libpython3.6m.so.1.0()(64bit), but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by python3-libs-3.6.8-62.el8_10.x86_64 from ubi-8-baseos-rpms
 Problem 7: package python3-libselinux-2.9-8.el8.x86_64 from ubi-8-baseos-rpms requires libpython3.6m.so.1.0()(64bit), but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by python3-libs-3.6.8-62.el8_10.x86_64 from ubi-8-baseos-rpms
 Problem 8: package gcc-toolset-13-runtime-13.0-2.el8.x86_64 from ubi-8-appstream-rpms requires /usr/sbin/semanage, but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by policycoreutils-python-utils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms
 Problem 9: package scl-utils-1:2.0.2-16.el8.x86_64 from ubi-8-appstream-rpms requires /usr/bin/modulecmd, but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by environment-modules-4.5.2-4.el8.x86_64 from ubi-8-baseos-rpms
 Problem 10: package python3-audit-3.1.2-1.el8.x86_64 from ubi-8-baseos-rpms requires python(abi) = 3.6, but none of the providers can be installed
  - package platform-python-3.6.8-62.el8_10.x86_64 from ubi-8-baseos-rpms requires libpython3.6m.so.1.0()(64bit), but none of the providers can be installed
  - package platform-python-3.6.8-62.el8_10.x86_64 from ubi-8-baseos-rpms requires python3-libs(x86-64) = 3.6.8-62.el8_10, but none of the providers can be installed
  - package platform-python-3.6.8-62.el8_10.i686 from ubi-8-appstream-rpms requires libpython3.6m.so.1.0, but none of the providers can be installed
  - package platform-python-3.6.8-62.el8_10.i686 from ubi-8-appstream-rpms requires python3-libs(x86-32) = 3.6.8-62.el8_10, but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by python3-libs-3.6.8-62.el8_10.x86_64 from ubi-8-baseos-rpms
  - nothing provides /usr/libexec/platform-python needed by python3-libs-3.6.8-62.el8_10.i686 from ubi-8-baseos-rpms
 Problem 11: package gcc-toolset-13-binutils-gold-2.40-21.el8.x86_64 from ubi-8-appstream-rpms requires gcc-toolset-13-runtime, but none of the providers can be installed
  - package gcc-toolset-13-runtime-13.0-2.el8.x86_64 from ubi-8-appstream-rpms requires /usr/sbin/semanage, but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by policycoreutils-python-utils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms
 Problem 12: package gcc-toolset-13-binutils-2.40-21.el8.x86_64 from ubi-8-appstream-rpms requires gcc-toolset-13-runtime, but none of the providers can be installed
  - package gcc-toolset-13-runtime-13.0-2.el8.x86_64 from ubi-8-appstream-rpms requires /usr/sbin/semanage, but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by policycoreutils-python-utils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms
 Problem 13: package gcc-toolset-13-gcc-13.2.1-6.2.el8.x86_64 from ubi-8-appstream-rpms requires gcc-toolset-13-runtime, but none of the providers can be installed
  - package gcc-toolset-13-runtime-13.0-2.el8.x86_64 from ubi-8-appstream-rpms requires /usr/sbin/semanage, but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by policycoreutils-python-utils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms
 Problem 14: package gcc-toolset-13-gcc-c++-13.2.1-6.2.el8.x86_64 from ubi-8-appstream-rpms requires gcc-toolset-13-runtime, but none of the providers can be installed
  - package gcc-toolset-13-runtime-13.0-2.el8.x86_64 from ubi-8-appstream-rpms requires /usr/sbin/semanage, but none of the providers can be installed
  - conflicting requests
  - nothing provides /usr/libexec/platform-python needed by policycoreutils-python-utils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms
 Problem 15: conflicting requests
  - package gcc-toolset-13-libstdc++-devel-13.2.1-6.2.el8.x86_64 from ubi-8-appstream-rpms requires gcc-toolset-13-runtime, but none of the providers can be installed
  - package gcc-toolset-13-libstdc++-devel-13.2.1-6.2.el8.i686 from ubi-8-appstream-rpms requires gcc-toolset-13-runtime, but none of the providers can be installed
  - package gcc-toolset-13-runtime-13.0-2.el8.x86_64 from ubi-8-appstream-rpms requires /usr/sbin/semanage, but none of the providers can be installed
  - nothing provides /usr/libexec/platform-python needed by policycoreutils-python-utils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms
 Problem 16: conflicting requests
  - package clang-libs-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64 from ubi-8-appstream-rpms requires gcc-toolset-13-gcc-c++, but none of the providers can be installed
  - package clang-libs-17.0.6-1.module+el8.10.0+20808+e12784c0.i686 from ubi-8-appstream-rpms requires gcc-toolset-13-gcc-c++, but none of the providers can be installed
  - package gcc-toolset-13-gcc-c++-13.2.1-6.2.el8.x86_64 from ubi-8-appstream-rpms requires gcc-toolset-13-runtime, but none of the providers can be installed
  - package gcc-toolset-13-runtime-13.0-2.el8.x86_64 from ubi-8-appstream-rpms requires /usr/sbin/semanage, but none of the providers can be installed
  - nothing provides /usr/libexec/platform-python needed by policycoreutils-python-utils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms
 Problem 17: conflicting requests
  - package clang-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64 from ubi-8-appstream-rpms requires libclang-cpp.so.17()(64bit), but none of the providers can be installed
  - package clang-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64 from ubi-8-appstream-rpms requires clang-libs(x86-64) = 17.0.6-1.module+el8.10.0+20808+e12784c0, but none of the providers can be installed
  - package clang-17.0.6-1.module+el8.10.0+20808+e12784c0.i686 from ubi-8-appstream-rpms requires libclang-cpp.so.17, but none of the providers can be installed
  - package clang-17.0.6-1.module+el8.10.0+20808+e12784c0.i686 from ubi-8-appstream-rpms requires clang-libs(x86-32) = 17.0.6-1.module+el8.10.0+20808+e12784c0, but none of the providers can be installed
  - package clang-libs-17.0.6-1.module+el8.10.0+20808+e12784c0.x86_64 from ubi-8-appstream-rpms requires gcc-toolset-13-gcc-c++, but none of the providers can be installed
  - package clang-libs-17.0.6-1.module+el8.10.0+20808+e12784c0.i686 from ubi-8-appstream-rpms requires gcc-toolset-13-gcc-c++, but none of the providers can be installed
  - package gcc-toolset-13-gcc-c++-13.2.1-6.2.el8.x86_64 from ubi-8-appstream-rpms requires gcc-toolset-13-runtime, but none of the providers can be installed
  - package gcc-toolset-13-runtime-13.0-2.el8.x86_64 from ubi-8-appstream-rpms requires /usr/sbin/semanage, but none of the providers can be installed
  - nothing provides /usr/libexec/platform-python needed by policycoreutils-python-utils-2.9-26.el8_10.noarch from ubi-8-baseos-rpms
lubomir commented 1 month ago

A module is basically a set of RPMs plus a yaml document. The RPMs have a special header flag that makes DNF refuse to install them if it doesn't have the yaml metadata (unless overridden).

The lockfile as written now contains URL to the RPMs, but doesn't include the metadata in any way. Additionally, cachi2 currently doesn't know anything about the modular metadata, and it would not download it anyway.

These steps would need to happen to make modules work:

Possible workarounds would be to include the module metadata in the repository manually, or setting module_hotfixes=1 option for the repo. The first option is fairly clean, the latter feels a lot like a hack.

msrb commented 1 month ago

@lubomir I am going to reveal my modularity ignorance here, but could you please be more specific about "include the module metadata in the repository manually"?

I know next to nothing about modules... where would we take the metadata from?

lubomir commented 1 month ago

If the repositories listed in the config file contains modular packages, it will also contain the metadata. It's an entry in the repodata with mdtype set to modules. For Brian's example in the first comment it's this file: https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/x86_64/appstream/os/repodata/b79ceccd138f06f9306192bcfe9cf22403d4507491c40589d945e2de89d1501a-modules.yaml.gz