NEXUS-43022: Group pypi repository for multiple proxy repositories is broken in 3.68.1

[yakirm-cr]

• What problem are you trying to solve? We are currently using Nexus OSS version 3.63 and we are trying to upgrade to a newer version (currently, 3.68.1). Unfortunately, Group pypi repository for multiple proxy repositories is broken in 3.68.1.

Here are the steps to reproduce the issue on Ubuntu 22.04 with docker:

1. Start Sonatype Nexus 3 via docker by running via terminal: docker run --rm -d --name nexus3 -p 8081:8081 sonatype/nexus3:3.68.1
2. Once Nexus is up (you can verify by running docker logs -f nexus3), login to it (you can obtain the admin password by running docker exec -ti nexus3 cat /nexus-data/admin.password) and follow the initial wizard.
3. Create the two proxy pypi repositories: i. pypi-proxy to https://pypi.python.org/. ii. piwheels-proxy to https://piwheels.org/.
4. Create a group pypi repository called pypi-all grouping the proxy repositories created in the previous step in the same order they are listed above - 1^st pypi-proxy and 2^nd piwheels-proxy.
5. Create a file called requirements.txt with the following context:

   attrs~=23.2.0
   bidict~=0.23.0
   cached-property~=1.5.2
   charset-normalizer~=3.3.0
   click~=8.1.3
   dnspython~=2.6.1
   fasteners~=0.18
   flasgger~=0.9.7.1
   flake8~=7.0.0
   Flask~=3.0.0
   Flask-Cors~=4.0.0
   Flask-SocketIO~=5.3.6
   gunicorn~=21.2.0
   importlib-metadata~=7.1.0
   importlib-resources~=6.4.0
   itsdangerous~=2.1.2
   Jinja2~=3.1.2
   jsonschema~=4.21.0
   MarkupSafe~=2.1.2
   mistune~=3.0.1
   netifaces~=0.11.0
   pip~=24.0
   pkgutil_resolve_name~=1.3.10
   pyrsistent~=0.20.0
   python-dateutil~=2.9.0
   python-engineio~=4.9.0
   python-socketio~=5.11.0
   PyYAML~=6.0
   referencing~=0.34.0
   requests~=2.31.0
   rpds-py~=0.18.0
   setuptools~=69.2.0
   six~=1.16.0
   urllib3~=2.2.0
   Werkzeug~=3.0.0
   zipp~=3.18.1

6. Run the following commands via terminal:

   python -m venv .env --prompt=test
   . .env/bin/activate
   pip install --no-cache --index-url=http://localhost:8081/repository/pypi-all/simple --trusted-host=localhost -r ./requirements.txt

   Expected result: All libraries listed in the requirments.txt are installed successfully in the Python virtual environment.

Actual results: Installation of one of the libraries (random) fails with the following error.

Collecting fasteners~=0.18
  Downloading http://localhost:8081/repository/pypi-all/packages/fasteners/0.19/fasteners-0.19-py3-none-any.whl (18 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    fasteners~=0.18 from http://localhost:8081/repository/pypi-all/packages/fasteners/0.19/fasteners-0.19-py3-none-any.whl#sha256=bba5c854998fca7d8b6acde63344d6a241b0228973455f307fddf55bfda24743 (from -r ./requirements.txt (line 7)):
        Expected sha256 bba5c854998fca7d8b6acde63344d6a241b0228973455f307fddf55bfda24743
             Got        758819cb5d94cdedf4e836988b74de396ceacb8e2794d21f82d131fd9ee77237

• Do you have a workaround you are using at present? Remain in version 3.63.

• What feature or behavior is this required for? Grouping multiple pypi repositories together as a single point of access.

• How could we solve this issue? (Not knowing is okay!) Unfortunately, I do not know. :slightly_frowning_face:

• Tell us about your Nexus Repository deployment: what version, operating system, and database are you using? We are running version 3.63 via Docker image sonatype/nexus3:3.63.0 (running on Ubuntu 22.04). In addition we are using the default internal OrientDB.

• Anything else? Nope.

[debug-richard]

I can confirm this issue with 3.68.0 and 3.68.1.
If a wheel is present in A and B the correct file from A is downloaded but the checksum from B is used.
It looks like the index got messed up, but rebuilding the indexes and invalidating the caches didn't help.
Since a downgrade from 3.68.0 is not possible, I would consider this an urgent issue.

[nblair]

Thanks @yakirm-cr and @debug-richard - we're looking at this and have setup an internal ticket to investigate.

[yakirm-cr]

@nblair This issue is blocking us to upgrade to version 3.68.1 (and above). Any update about it? Please, advise.