search

pulp / pulp_rpm

RPM support for Pulp Platform
https://pulpproject.org/pulp_rpm/
GNU General Public License v2.0
48 stars 124 forks source link

Repositories only containing subrepos (no repodata folder on repository root) are not supported #3452

Open CERN-LHCb-Online opened 6 months ago

CERN-LHCb-Online commented 6 months ago

Version pulpcore version: 3.47.0 pulp_rpm version: 3.25.1 Installed with pulp-minimal:3.47.0 docker container

Describe the bug RHEL9 installation dvds have the following structure : 

ls -la
total 53
dr-xr-xr-x 1root root  2048 27 oct.   2022 .
drwxr-xr-x 3 root root  4096  8 juil.  2020 ..
dr-xr-xr-x 1 root root  2048 27 oct.   2022 AppStream
dr-xr-xr-x 1 root root  2048 27 oct.   2022 BaseOS
-r--r--r-- 1 root root    60 27 oct.   2022 .discinfo
dr-xr-xr-x 1 root root  2048 27 oct.   2022 EFI
-r--r--r-- 1 root root  8154 27 oct.   2022 EULA
-r--r--r-- 1 root root  1455 27 oct.   2022 extra_files.json
-r--r--r-- 1 root root 18092 27 oct.   2022 GPL
dr-xr-xr-x 1 root root  2048 27 oct.   2022 images
dr-xr-xr-x 1 root root  2048 27 oct.   2022 isolinux
-r--r--r-- 1 root root   103 27 oct.   2022 media.repo
-r--r--r-- 1 root root  1669 27 oct.   2022 RPM-GPG-KEY-redhat-beta
-r--r--r-- 1 root root  3682 27 oct.   2022 RPM-GPG-KEY-redhat-release
-r--r--r-- 1 root root  1560 27 oct.   2022 .treeinfo

If we define this as a remote, we cannot currently do a sync as the sync immediatly stops trying to access a repodata/repomd.xml file as the root of the repository.

pulp --config /admin/.private/pulp.ini -v rpm repository sync --name RHEL9EB_BaseOS --sync-policy mirror_complete
repositories_rpm_rpm_list : get http://pulp:8080/pulp/api/v3/repositories/rpm/rpm/?name=RHEL9EB_BaseOS&offset=0&limit=1
Response: 200
repositories_rpm_rpm_sync : post http://pulp:8080/pulp/api/v3/repositories/rpm/rpm/018de9c3-229f-793c-9bb3-cc2861832657/sync/
Response: 202
tasks_read : get http://pulp:8080/pulp/api/v3/tasks/018de9c3-73f1-7eaa-bb64-e112852939b3/
Response: 200
Started background task /pulp/api/v3/tasks/018de9c3-73f1-7eaa-bb64-e112852939b3/
Error: Task /pulp/api/v3/tasks/018de9c3-73f1-7eaa-bb64-e112852939b3/ failed: 'An invalid remote URL was provided: http://10.128.124.12:8000/rhel-baseos-9.1-x86_64/'

On the http server, we get the following error : 

10.128.124.142 - - [27/Feb/2024 09:51:57] code 404, message File not found
10.128.124.142 - - [27/Feb/2024 09:51:57] "GET /rhel-baseos-9.1-x86_64/repodata/repomd.xml HTTP/1.1" 404 -
10.128.124.142 - - [27/Feb/2024 09:51:57] "GET /rhel-baseos-9.1-x86_64 HTTP/1.1" 301 -
10.128.124.142 - - [27/Feb/2024 09:51:57] "GET /rhel-baseos-9.1-x86_64/ HTTP/1.1" 200 -

To Reproduce

Expected behavior Complete sync of the the installation repository, following the definition in the .treeinfo file.

Additional context I may have time in few months to make a test setup and try to fix this issue, but not in March

CERN-LHCb-Online commented 6 months ago

The AlmaLinux-9-latest-x86_64-minimal.iso image also have no repodata folder at the root. It only have a 'Minimal' variant.

CERN-LHCb-Online commented 6 months ago

In the mean time, I moved the package & repodata folders out of the BaseOS folder and modified the .treeinfo file accordingly and was then able to do a sync.

dralley commented 6 months ago

Thank you for the report!

Closing https://github.com/pulp/pulp_rpm/issues/2306 as a duplicate of this one (which has more info available).

We haven't considered the possibility of a repository having a .treeinfo but no top-level repodata. We need to investigate whether this breaks anything (for example: autopublish), and potentially add a new test fixture to exercise it.

I don't think we had a concrete example of such a repository (or how the scenario would arise) previously.