Open joekiller opened 6 months ago
dimbleby
commented
6 months ago Duplicate #9293 with - of course - the same answer. In particular see comment about removing the JSON API, which would remove the difference.
https://github.com/python-poetry/poetry/issues/9293#issuecomment-2048235447
Bottom line: if repositories provide bad data, resolvers will find bad answers.
Please close.
joekiller
commented
6 months ago I attributed this directly to the legacy resolver as the legacy resolver installed distutils sdist labeled as 0.21.post1 even though the metadata within the package said 0.21. As it stands, poetry will install packages with inconsistencies vs what the payload was expected to contain at minimum with the legacy installing making this bug applicable and unresolved as I do not see a duplicate raising this specific issue.
I cannot immediately tell from reviewing the tests and code itself it poetry is guarding from installing packages with metadata that does not match the expected metadata of what has been requested to be installed.
If it is guarding could you point to the functional area where this is covered? That reference would offer myself and anyone else looking into this edge case reassurance that poetry will not install packages with inconsistent metadata if #9170 is merged.
Pip, for example, has explicit tests and errors that it throws when inconsistencies are encountered. These inconsistencies are the type that come up when the payload of the targeted package do not meet the expected metadata presented by the API. Just as the warehouse had proven, the contents of the package behind the URL should not be blindly trusted even if the assurances of the new means of downloading it are met.
The PR #9170 mentions that test_solver_skips_invalid_versions() is deliberately removed which makes me think that there is potential for payloads that are not what they seem to be allowed to slip through. Could a test similar to pip's test_new_resolver_skip_inconsistent_metadata or just the test_solver_skips_invalid_versions be added back?
dimbleby
commented
6 months ago test_solver_skips_invalid_versions() is completely irrelevant, is not removed from main, and is not removed in that pull request anyway.
joekiller
commented
6 months ago It seems that poetry will install packages whose metadata doesn't match what is expected.
Poetry did install a package whose metadata didn't match what was expected.
Is this behavior considered a defect?
dimbleby
commented
6 months ago to me, this is not interesting. People can put any old rubbish into their distribution if they want to: it is the responsibility of package builders to build correct packages.
if you care to do anything about this then you probably want to start somewhere near PackageInfo.from_wheel() and PackageInfo.from_sdist()
Description
Doing analysis related to the docutils 0.21 release snafu, see https://github.com/python-poetry/poetry/issues/9293, it appears the Legacy Repository Handler allows packages of incorrect metadata to be installed. I cannot speak to if the pypi repository handler is allowing this as this is piggybacking off my #9293 analysis.
If the repository handler would assert the metadata version vs what it retrieved the divergent repo paths would converge at the same conclusion, of "bad package".
Having a colleague simply utilize a renamed pypi and having success to install docutils 0.21 was quite strange. I get that the alternative naming was causing a legacy path and that there's a lot to juggle here and address this at warehouse is helpful to prevent such snafu. There is the https://github.com/pypi/warehouse/issues/15749 post regarding enforcing naming of the package better however the metadata within the package was still 0.21. Reviewing docutils bugs 483 it seems pip will reject the package when the metadata is incorrect.
Not to beat on docutils too hard here, but it seems that they had a history of these types of meta changes.
https://pypi.org/project/docutils/0.15/#files
This package seems to have hit a corner uncleaned for a very long time. Perhaps it'll add some imperative to resolve legacy behaviors.
Should poetry be checking the package metadata consistently or is this expected behavior?
Workarounds
Not really a work around but I think this alternative path is the explanation for a common theme of confusion where people specify a non-prioritized pypi repository via something like:
and then experience "when I create a different pypi repo my install works". Combined with the fact that warehouse can allow some inappropriate package naming to take place, it seems this alternative path makes some installations work.
Making poetry consistently check the package metadata and alert the user, regardless of the repository handler path may be a way to reduce some of the confusion.
Poetry Installation Method
pip
Operating System
mac os 14.4.1
Poetry Version
1.8.2
Poetry Configuration
Python Sysconfig
Example pyproject.toml
Poetry Runtime Logs