dimbleby
commented
8 months ago please remove all the unnecessary parts from the pyproject.toml, and provide a series of commands to reproduce a problem.
Open opcode81 opened 8 months ago
dimbleby
commented
8 months ago please remove all the unnecessary parts from the pyproject.toml, and provide a series of commands to reproduce a problem.
opcode81
commented
8 months ago @dimbleby, the relevant part of the pyproject.toml is mentioned in the issue description: a dependency which uses a git commit hash and where the hash is later changed. (I only pasted the full file, because your workflow asked for it.)
I think it would make the most sense to analyse this issue by looking at the source code which resolves git-based dependencies - as there is clearly a resolution problem which, under certain conditions, makes Poetry use the wheel associated with another hash (which is in the cache alongside the newly built wheel).
If you really wanted to reproduce it, it would involve steps where the pyproject.toml is changed to a new hash, but it may or may not be deterministic. See issue description.
dimbleby
commented
8 months ago One might get lucky reading source code - you are very welcome to try it - but I don't like my chances
A minimal reproducible example is much the most helpful thing that an issue report can provide: yes, please.
opcode81
commented
8 months ago I couldn't reproduce it just now with a minimal example. But I am also not surprised, because out of our team of 7 developers, only 2 people experienced the issue. Others experienced it the last time we changed the commit hash.
bavalpey
commented
8 months ago One might get lucky reading source code - you are very welcome to try it - but I don't like my chances
A minimal reproducible example is much the most helpful thing that an issue report can provide: yes, please.
It's a bit odd to ask for a MRE when the issue is nondeterministic.
opcode81
commented
7 months ago @dimbleby I am providing instructions and resources for reproduction in this repository: https://github.com/opcode81/poetry_issue_9075
dimbleby
commented
7 months ago not non-deterministic after all then?
I haven't run your repro but if the suspicion is that a cache for revision abc is being wrongly reused for revision def - you'll want to look at this code.
The directory name for the cached archive for git dependencies should have incorporated the exact revision so that's supposed to work out just fine - add some trace and dig into where that's going wrong,
opcode81
commented
7 months ago I asked the last person who ran into the issue to archive their cache, and that's the basis for the reproduction. Unfortunately, it's still unclear which conditions led to the cache being constructed the way it is in the repo I provided. It happened on 3 of our machines but not all of them.
Unfortunately, I do not have time to look into fixing the issue myself; we have since decided to switch from Poetry to conda/mamba. There were just way too many problems that were wasting time we could have better spent making some actual progress in our project.
dimbleby
commented
7 months ago Unfortunately, it's still unclear which conditions led to the cache being constructed the way it is
fair enough, but without a way to reproduce this I expect there is no way to do anything useful with it.
If you have given up on this, might as well close it: assuming it is real then someone will eventually show up with a way to reproduce it.
opcode81
commented
7 months ago With the repo I provided, you can at least partially reproduce the issue: You can analyse the dependency resolution and determine what is causing Poetry to pick an old, cached version instead of the correct one. Did you do this? And is it already clear that without the process that led to the cache being created in this way, there is no chance of knowing what the actual problem is? I didn't get the impression that this analysis has already happened, and yet you want to close this. Are you not interested in getting to the bottom of this?
I went to great lengths in order to provide the Poetry developers with detailed information on this issue as well as a repository for reproduction - and the response I get is that if I'm not going to look into it myself, you're just going to close this? I really don't understand why the default is, apparently, to do nothing at all if there is not a way to reproduce the issue from scratch.
What I provided should be a sufficient basis to fix the problem. Two scenarios:
opcode81
commented
7 months ago The fact that adding --no-cache to the Poetry command does not fix the issue, as originally reported, is also a bug that should be looked into.
dimbleby
commented
7 months ago >>> from poetry.utils.cache import ArtifactCache
>>> from pathlib import Path
>>> c = ArtifactCache(cache_dir=Path(""))
>>> c.get_cache_directory_for_git("https://github.com/thu-ml/tianshou.git", "9b6cb6903eddb529437f949b76ffa87a9b9115e9", subdirectory=None)
PosixPath('76/5c/9c/8778c975cd921e5de6d822642d4fbfcccf57595fde67eec516dfcba17d')So I expect that the wheel in that place in your cache is the one that poetry uses
does the wheel in that place in your repro correpond to that revision or not?
I expect "no", in which case the question is: so how did it get there? (what does it correspond to?) and then we have to go back to you to provide a way to reproduce this
I am fairly sure that you are the person in the world who has the most interest in getting to the bottom of this. If you are now not using poetry therefore don't care... well I suppose it makes little difference whether we all ignore an open issue or all ignore a closed issue, my tidy-minded preference is to close such things.
opcode81
commented
7 months ago does the wheel in that place in your repro correpond to that revision or not?
It does not.
how did it get there?
It's hard to say. Since the directory is determined by hashing the URL and revision with SHA-256, we can basically rule out a hash collision. But perhaps there is an issue with the cloning mechanism where it's possible for an old cached checkout (in the cache's src/ folder) to be used instead of a fresh checkout. I had a brief look at the respective source code, and while poetry does try to reuse an existing checkout, I found no obvious issues.
what does it correspond to?
I can't pinpoint it exactly, but it appears to be a version from around November 26, 2023, whereas the requested revision is from February 15, 2024.
it makes little difference whether we all ignore an open issue or all ignore a closed issue
I strongly disagree. Closed issues are less easily discovered, especially on GitHub, where the default search filter contains "is:open". If someone runs into the same issue, they may not find this issue; it is unexpected for an unresolved issue to be closed. Likewise, the poetry developer who decides to look into the then newly created issue may not find this issue either and will miss out on the insights that have already been gathered. I would close an issue only if it was either resolved or deemed a non-issue. Unless you think I am making this up, neither should be the case here.
And you still haven't commented on the ineffectiveness of --no-cache.
lorenzwalthert
commented
1 month ago I had the same problem. Clearing the cache did not do the trick for me. However, probably that was because I configured poetry to create a .venv folder inside the project directory... Once I did that too, things worked (around). Posting this for people who might have the same configuration.
Description
In our project, we depend on the source version of the tianshou library like so:
We update the commit hash sometimes, while the version that is stored in tianshou's pyproject.toml remains the same (it has been 0.5.1 for almost a year).
Now the issue that we encounter is that after the commit hash has been changed, on some machines, the new version of tianshou will not get installed when calling poetry install (after a lock update). Poetry will report this
but will in fact install a different version that is also associated with 0.5.1 but has a different hash. The exact conditions under which it will install an incorrect version are unknown. (It appears random to us.)
The issue only occurs on machines that had previously installed a different hash (but which is also 0.5.1).
The issue cannot be fixed by calling poetry with
--no-cache, as this will not result in the tianshou repo being re-cloned! (Which I think is another, unrelated bug.) The issue can be fixed by deleting all cached versions of tianshou from the poetry cache.Workarounds
Delete all cached versions of the respective dependency from the poetry cache. On Ubuntu find all files starting with
tianshouin ~/.cache/pypoetry and delete the containing folders.Poetry Installation Method
pipx
Operating System
Ubuntu
Poetry Version
1.7.1
Poetry Configuration
Python Sysconfig
Example pyproject.toml
Poetry Runtime Logs