search

frostming / unearth

A utility to fetch and download python packages
https://unearth.readthedocs.io
MIT License
141 stars 18 forks source link

PyPI (pypi.org) gets added as an index even though it's overwritten in PDM's pyproject.toml #95

Open MikulasZelinka opened 10 months ago

MikulasZelinka commented 10 months ago

Describe the bug PyPI (pypi.org) gets added as an index even though it's overwritten in pyproject.toml using the pypi name in unearth>=0.13

I guess it could be caused by https://github.com/frostming/unearth/commit/7d1c215 (although I don't really see how it's possible looking at the source code)...

Version 0.12.1 works fine and does not have this issue.

I'm not sure if this is a pdm or an unearth issue.

To Reproduce

[[tool.pdm.source]]
name = "pypi"
url = "https://<some_custom_artifactory_url>/artifactory/api/pypi/pypi-all/simple"
verify_ssl = true
include_packages = ["*"]
pdm install -vv --prod --no-lock --check --no-editable --no-self --fail-fast

Now the log of install looks like:

18:06:22  #12 2.285 Run command: ['/usr/local/bin/python3.11', '-m', 'virtualenv', '/<package>/.venv', '-p', '/usr/local/bin/python', '--prompt=<package>-3.11', '--no-pip', '--no-setuptools', '--no-wheel']
18:06:22  #12 2.****78 created virtual environment CPython3.11.6.final.0-6**** in 89ms
18:06:22  #12 2.****78   creator CPython3Posix(dest=/<package>/.venv, clear=False, no_vcs_ignore=False, global=False)
18:06:22  #12 2.****78   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
18:06:22  #12 2.****99 Virtualenv is created successfully at /<package>/.venv
18:06:32  #12 11.62 STATUS: Fetching hashes for resolved packages...
18:06:32  #12 11.67 Synchronizing working set with resolved packages: 100 to add, 0 to update, 0 to remove
18:06:32  #12 11.67 
18:06:32  #12 11.88 unearth.collector: Collecting links from https://pypi.org/simple/anyio/
... (repeats many times for different packages)
18:06:32  #12 11.89 unearth.auth: Found credentials in index url for pypi.org
18:06:33  #12 13.00 unearth.collector: Fetching HTML page https://pypi.org/simple/annotated-types/
18:06:33  #12 13.00 unearth.collector: Collecting links from https://<some_custom_artifactory>/artifactory/api/pypi/pypi-all/simple/annotated-types/
... (artifactory links are mixed in for the packages too, but in the end downloads from pypi are preferred)

Expected behavior The only index should be the some_custom_artifactory_url. But instead, both PyPI and this custom index are used.

It also looks like PyPI is preferred, and the custom artifactory is used iff a package is not found in PyPI.

System (please complete the following information):

pdm is installed using these steps from https://pdm-project.org/latest/usage/advanced/#use-pdm-in-a-multi-stage-dockerfile

When installing like this, this issue appears:

pip install -U pip setuptools wheel
pip install pdm

And the issue disappears if I fix the unearth version:

pip install -U pip setuptools wheel
pip install pdm unearth==0.12.1

Additional context Will try to provide more details if necessary. It's a bit difficult to provide an MRE in this case as there any many components at play. In fact, this issue appears inside a Docker build job running inside a Jenkins pipeline.

MikulasZelinka commented 10 months ago

Looks like PDM simply doesn't pass the indexes to unearth?

https://github.com/pdm-project/pdm/blob/7abdbcc1d33251f7f8dc06bd7d8cb8e56e78028e/src/pdm/environments/base.py#L154C5-L166

Let me know if I should make an issue in PDM instead.

A quick fix would probably be to specify ==0.12.1 here: https://github.com/pdm-project/pdm/blob/7abdbcc1d33251f7f8dc06bd7d8cb8e56e78028e/pyproject.toml#L21

MikulasZelinka commented 10 months ago

Possibly fixed by https://github.com/pdm-project/pdm/pull/2563