Secrus
commented
2 weeks ago Having 2 primary sources is wrong (it should be caught at an earlier stage I think) and that is most likely why you see this issue.
Closed vit-zikmund closed 2 weeks ago
Secrus
commented
2 weeks ago Having 2 primary sources is wrong (it should be caught at an earlier stage I think) and that is most likely why you see this issue.
vit-zikmund
commented
2 weeks ago Hi @Secrus, what's the issue with multiple primary sources? The docs say that's perfectly fine and we've been using them for ages.
... by adding at least one primary source ... ... All primary package sources are searched for each dependency ...
radoering
commented
2 weeks ago The docs say that's perfectly fine and we've been using them for ages.
Yes, it should be fine. (You can only have one default source, but default is deprecated anyway.)
vit-zikmund
commented
2 weeks ago Thanks @radoering. FWIW here's a couple short instructions how to setup & run a devpi server for a local reproduction, unless you have it, duh :innocent::
mkdir devpi && cd devpi
python -m venv . && source bin/activate
pip install devpi-server devpi-client
devpi-init
devpi-server
# the server should be running now, setup an empty test index
devpi use http://localhost:3141/
devpi login --password '' root
devpi index -c root/testNow change to url of the sources to:
[[tool.poetry.source]]
name = "mama-prod"
url = "http://localhost:3141/root/test/+simple/"
priority = "primary"
[[tool.poetry.source]]
name = "mama-pypi-mirror"
url = "http://localhost:3141/root/pypi/+simple/"
priority = "primary"Running poetry update in a fresh env should reproduce the behavior.
radoering
commented
2 weeks ago Unfortunately, I cannot reproduce the issue with this setup. I even cleared my cache (including artifacts). The installation always succeeds on my machine. I also debugged into RepositoryPool.package() and always get a PackageNotFoundError for the first source.
This is easily triggered with the right config and the absence of
poetry.lock.
If it cannot be triggered if poetry.lock exists, it should be possible to narrow the issue down to poetry lock and find something wrong in poetry.lock. If this is not a locking issue but an installation issue, it should not matter if poetry.lock exists or not.
I think the problem might be the
CachedRepository.package()call, which should return the exception, but it doesn't seem to even check if it contains the package and seems to go straight to creating one 🤔
This should finally call LegacyRepository._get_release_info(), which calls LegacyRepository._get_page(), which raises the exception.
Maybe, your cache is corrupt. Did you try --no-cache / clearing your cache?
vit-zikmund
commented
2 weeks ago Hi @radoering, using --no-cache indeed fixes the issue :O
I haven't yet deleted the cache, are there some artifacts I could collect to help you see why/how the cache got corrupted? I will understand that might be under your radar :)
This might be a good suggestion for a bug template, as clearing the cache is something I haven't thought about to try.
radoering
commented
2 weeks ago I haven't yet deleted the cache, are there some artifacts I could collect to help you see why/how the cache got corrupted?
It is difficult to name the exact artifacts because the cache is structured by hashes. Further, we will only see what is corrupt not how it got there. The most likely reason I can think of is that mama-prod had been configured to forward to mama-pypi-mirror when the cache was created or that sources were renamed so that the packages were available in mama-prod in the past.
This might be a good suggestion for a bug template, as clearing the cache is something I haven't thought about to try.
I agree.
vit-zikmund
commented
2 weeks ago Well, those are interesting possibilities. I can't say I haven't done such fiddling while figuring some CI/CD things out. Thanks a ton for your time and I'm sincerely sorry it's such a bummer.
Filed a proper issue (with a perfect number :imp:) for the template improvement.
There's nothing else to do here. Thank you for your help!
Description
In case there are two primary sources with the first being a private one (having a couple custom packages only) and another source mirroring/being PyPI (having it all), some of the packages from PyPI get attempted to download from the 1st source despite
LegacyRepository._find_packagesproperly finding them in the right source.This is easily triggered with the right config and the absence of
poetry.lock.I've tried to chase this down on the latest
main(same behavior), but no dice. However, I got some clues:The package resolving around:
works well, I've put some prints here and there and I see the
Package.source_urlthat's coming out ofLegacyRepository._find_packagesis correct and has the right url. The problem happens inVersionSolver._choose_package_version. Before the call toProvider.complete_package, the package has a corectsource_urland after the call it's wrong. Inside the method, the problematic place is this call toRepositoryPool.package(), where it iterates the repositories, touching first the one not having the package here. Instead of raising aPackageNotFoundexception, theLegacyRepository.package()creates one out of thin air with the wrongsource_url. I think the problem might be theCachedRepository.package()call, which should return the exception, but it doesn't seem to even check if it contains the package and seems to go straight to creating one :thinking:I confess I didn't get the surrounding semantics, so maybe that's not the place, but it seems pretty suspicious.
Workarounds
Put the public source to the first place and the private second, but this likely works only by accident.
Poetry Installation Method
pipx
Operating System
Fedora 39
Poetry Version
Poetry (version 1.8.3)
Poetry Configuration
Python Sysconfig
No response
Example pyproject.toml
Poetry Runtime Logs