hlin commented 2 months ago

DNF requires the module metadata in repodata to install modular packages, so that if there's any modular packages in the lock file, the module metadata file should be included too.

https://github.com/konflux-ci/rpm-lockfile-prototype/pull/33 generates the lock file with a new module_metadata field and then cachi2 should download the metadata files together with RPMs and createrepo_c can recognize the metadata files when generating repository.

---
lockfileVersion: 1
lockfileVendor: redhat
arches:
- arch: x86_64
  packages: [...]
  source: [...]
  # ↑ already exists, ↓ is new
  module_metadata:
  - url: <URL>                  # needed to know what to download
    repoid: <repoid>            # need to know where to put the file
    size: <size>                # integrity checking
    checksum: sha256:<checksum> # integrity checking

Maintainers will complete the following section

[ ] Commit messages are descriptive enough
[ ] Code coverage from testing does not decrease and new code is covered
[ ] Docs updated (if applicable)
[ ] Docs links in the code are still valid (if docs were updated)

Note: if the contribution is external (not from an organization member), the CI pipeline will not run automatically. After verifying that the CI is safe to run:

approve GitHub Actions workflows by clicking a button
approve the Red Hat Trusted App Pipeline container build by commenting /ok-to-test (as is the standard for Pipelines as Code)

hlin commented 2 months ago

With this patch, createrepo can recognize modules.yaml.gz and mdtype field (and modifyrepo) is not used, @lubomir can you take a look?

lubomir commented 2 months ago

With this patch, createrepo can recognize modules.yaml.gz and mdtype field (and modifyrepo) is not used, @lubomir can you take a look?

That's a good point. Createrepo will match on the file name. Realistically the URL will point to some existing repodata, where the filename will match what createrepo expects. The mdtype in the lockfile is indeed not necessary, so cachi2 could drop it.

eskultety commented 2 months ago

I confirm that this approach which is consistent with the rest of the lockfile based logic and behaviour works. I tested this with the following minimalistic lockfile and Fedora 38 (39+ dropped modularity):

---
lockfileVersion: 1
lockfileVendor: redhat
arches:
- arch: x86_64
  packages:
  - url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/38/Modular/x86_64/os/Packages/s/subversion-libs-1.14.2-5.module_f38+15272+edcc0b97.x86_64.rpm
    repoid: fedora-modular
    size: 1583542
    checksum: sha256:b52f95619aa4ee70190fce7837985504550938fe0c0ebfba0f2ccd94b9a0a897
    name: subversion-libs

  - url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/38/Everything/x86_64/os/Packages/a/apr-1.7.2-2.fc38.x86_64.rpm
    repoid: fedora
    name: apr
    size: 130140
    checksum: sha256:ca4680178a8d7d5562dfbd361bc7caa13f8fc9a2756d337cf9ded77687e763ee

  - url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/38/Everything/x86_64/os/Packages/a/apr-util-1.6.3-2.fc38.x86_64.rpm
    name: apr-util
    repoid: fedora
    size: 97926
    checksum: sha256:ecad94eb8493689ac063df1d4e141ea992996d1a1a73f7a6b77724d93b4063c0

  - url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/38/Everything/x86_64/os/Packages/l/libserf-1.3.9-27.fc38.x86_64.rpm
    name: libserf
    repoid: updates
    size: 57965
    checksum: sha256:d04afa98de14d4a30ad8f4c335f0fa458ec89596a5bf6848718989ff03755363

  - url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/38/Everything/x86_64/os/Packages/u/utf8proc-2.7.0-4.fc38.x86_64.rpm
    name: utf8proc
    repoid: fedora
    size: 82281
    checksum: sha256:5b4cdd58179acda1cbeedfbacba872d9e43f39c629486860420cf578b30c9a2c

  metadata:
  - url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/38/Modular/x86_64/os/repodata/a5b951a984ab6b7edd2c08d68162014d9f6ccfdbc4c721d318edca0a21697cb1-modules.yaml.gz
    repoid: fedora-modular
    mdtype: modules
    size: 31546
    checksum: sha256:a5b951a984ab6b7edd2c08d68162014d9f6ccfdbc4c721d318edca0a21697cb1

Then, after having run fetch --dev-package-mangers rpm and inject-files cachi2-output --for-output-dir /tmp/

$ podman run -it --rm --network=none -v cachi2-output/deps/rpm/x86_64/repos.d/:/etc/yum.repos.d/:rw,Z -v cachi2-output/deps:/tmp/deps:rw,Z fedora:38 /bin/bash

# dnf module enable -y subversion:1.14
==========================================================================================================
 Package                  Architecture            Version                  Repository                Size
==========================================================================================================
Enabling module streams:
 subversion                                       1.14                                                   

Transaction Summary
==========================================================================================================
Complete!

# dnf install subversion-libs -y
Repository 'fedora-modular' is missing name in configuration, using id.
Repository 'fedora' is missing name in configuration, using id.
Repository 'updates' is missing name in configuration, using id.
Last metadata expiration check: 0:00:06 ago on Wed Aug 28 11:28:29 2024.
Dependencies resolved.
==========================================================================================================
 Package               Architecture Version                                    Repository            Size
==========================================================================================================
Installing:
 subversion-libs       x86_64       1.14.2-5.module_f38+15272+edcc0b97         fedora-modular       1.5 M
Installing dependencies:
 apr                   x86_64       1.7.2-2.fc38                               fedora               127 k
 apr-util              x86_64       1.6.3-2.fc38                               fedora                96 k
 libserf               x86_64       1.3.9-27.fc38                              updates               57 k
 utf8proc              x86_64       2.7.0-4.fc38                               fedora                80 k

Transaction Summary
==========================================================================================================
Install  5 Packages

Total size: 1.9 M
Installed size: 5.6 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                  1/1 
  Installing       : apr-1.7.2-2.fc38.x86_64                                                          1/5 
  Installing       : apr-util-1.6.3-2.fc38.x86_64                                                     2/5 
  Installing       : libserf-1.3.9-27.fc38.x86_64                                                     3/5 
  Installing       : utf8proc-2.7.0-4.fc38.x86_64                                                     4/5 
  Installing       : subversion-libs-1.14.2-5.module_f38+15272+edcc0b97.x86_64                        5/5 
  Running scriptlet: subversion-libs-1.14.2-5.module_f38+15272+edcc0b97.x86_64                        5/5 
  Verifying        : subversion-libs-1.14.2-5.module_f38+15272+edcc0b97.x86_64                        1/5 
  Verifying        : apr-1.7.2-2.fc38.x86_64                                                          2/5 
  Verifying        : apr-util-1.6.3-2.fc38.x86_64                                                     3/5 
  Verifying        : utf8proc-2.7.0-4.fc38.x86_64                                                     4/5 
  Verifying        : libserf-1.3.9-27.fc38.x86_64                                                     5/5 

Installed:
  apr-1.7.2-2.fc38.x86_64               apr-util-1.6.3-2.fc38.x86_64                                      
  libserf-1.3.9-27.fc38.x86_64          subversion-libs-1.14.2-5.module_f38+15272+edcc0b97.x86_64         
  utf8proc-2.7.0-4.fc38.x86_64         

Complete!

hlin commented 1 month ago

Can someone help to trigger the workflows?

hlin commented 1 month ago

Are there any more blockers to merge this PR?

hlin commented 1 month ago

We have a target end date of Sep 11 for this feature. Can this PR be merged then?

eskultety commented 1 month ago

We have a target end date of Sep 11 for this feature. Can this PR be merged then?

Release is planned around 17th-18th, we'll get it in by then. @brunoapimentel or @taylormadore can we get one more ACK here?

brunoapimentel commented 1 month ago

/ok-to-test

containerbuildsystem / cachi2

package_managers: rpm: Support prefetching modules metadata #610

Maintainers will complete the following section