`poetry cache clear` should be able to clear all information for a single package

binh-vu commented 3 years ago

[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] I have searched the documentation and believe that my question is not covered.

Feature Request

Thanks for making an awesome product.

In my project, I split my code to several packages, which are published to pypi. The problem is after I upload a new version of a package on pypi, poetry doesn't know that the package has been updated and throw SolverProblemError (depends on package X which doesn't match any versions) if I use the updated package somewhere else.

I think this may be due caching. Clearing poetry cache: poetry cache clear pypi --all does help me solve the problem. However, it's so slow as poetry has to repopulate the cache for all other packages.

I think it would be great to have:

the ability to invalidate/clear cache of specific packages, or
automatically fetch new version of a package if the requested version is newer than the current one in the cache or when it fails to resolve dependencies.

Thanks!

dusktreader commented 2 years ago

This would be super useful for package developers. I frequently publish my own packages to pypi and then want to update another project with the new dependency. However, there's no way for me to tell the cache to ignore a specific package. So, I get errors like this frequently:

$ poetry update
Updating dependencies
Resolving dependencies... (0.3s)

  SolverProblemError

  Because armasec depends on snick (^1.3.0) which doesn't match any versions, version solving failed.

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Now, I know for a fact that snick==1.3.0 is available, because I just published it and can see it on pypi. Further, if I create a new venv and pip install snick, it gets version 1.3.0.

The only workaround available I can see is to dump the whole cache for pypi, and it makes me sad to lose all those cached packages when it's just the one where I need to bypass the cache.

If there was a way to purge a single package from the cache, it would be extremely helpful.

dusktreader commented 2 years ago

After digging into the source, it turns out you can delete an entry for a specific version of a package:

$ poetry cache clear pypi:snick:1.2.0

However, this will not help you in this case because Poetry is using the cached package info: https://github.com/python-poetry/poetry/blob/master/src/poetry/repositories/pypi_repository.py#L204

So, I think the cache clear command needs a way to tell it to purge the package info from the cache instead.

May tinker with this a bit and see if I can throw together a PR for this functionality.

dusktreader commented 2 years ago

Oh, and the documentation and --help needs to be updated to let folks know about clearing specific package versions.

jedie commented 2 years ago

The problem is after I upload a new version of a package on pypi, poetry doesn't know that the package has been updated

I confirm this (Poetry v1.1.14), because i have the same trouble again and again. e.g.:

~/repos/pyinventory_ynh$ poetry update 
Updating dependencies
Resolving dependencies... (4.9s)

  SolverProblemError

  Because pyinventory-ynh depends on pyinventory (>=0.14.0rc1) which doesn't match any versions, version solving failed.

  at ~/.local/lib/python3.9/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Poetry didn't find it

~/repos/pyinventory_ynh$ poetry update -vvv
...
PyPI: 0 packages found for pyinventory >=0.14.0rc1
   1: fact: no versions of pyinventory match >=0.14.0rc1
   1: conflict: no versions of pyinventory match >=0.14.0rc1
   1: !  pyinventory (>=0.14.0rc1) is satisfied by  pyinventory (>=0.14.0rc1)
   1: ! which is caused by "pyinventory-ynh depends on pyinventory (>=0.14.0rc1)"
   1: ! thus: version solving failed
   1: Version solving took 0.182 seconds.
   1: Tried 1 solutions.
...

pip found it:

~/repos/pyinventory_ynh$ poetry run pip install "pyinventory>=0.14.0rc1"
Collecting pyinventory>=0.14.0rc1
  Downloading PyInventory-0.14.0rc1-py3-none-any.whl (120 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.5/120.5 kB 4.9 MB/s eta 0:00:00
...
Installing collected packages: pyinventory
  Attempting uninstall: pyinventory
    Found existing installation: pyinventory 0.13.0
    Uninstalling pyinventory-0.13.0:
      Successfully uninstalled pyinventory-0.13.0
Successfully installed pyinventory-0.14.0rc1

It exists: https://pypi.org/project/pyinventory/0.14.0rc1/ Note: it's not related to the version scheme: I tested 0.14.0rc1 and 0.14.0.rc1 and 0.14.0rc2 (This is just a example. I see the same behavior in other constellation, after a new release published to PyPi)

rm -Rf ~/.cache/pypoetry/cache/ helps here, sometimes, but often not.

poetry run pip install -U <PackageName> will often install the version i need, but poerty didn't find it. So i assume it's there and public on PyPi.

Also boring: The error message like Because <foo> depends on <bar> (>=0.14.0rc1) which doesn't match any versions, version solving failed. if not very helpfull :(

pipenv has a little bit better error message, because it lists all found versions.

EDIT: Adding allow-prereleases = true (see also: https://github.com/python-poetry/poetry/issues/4405 ) doesn't help here, too.

EDIT2: Maybe duplicated by https://github.com/python-poetry/poetry/issues/3543 ?

tony commented 2 years ago

I'm getting hit by this right now 😬. Now the old workaround in 1.1.x of wiping the cache no longer works.

Side note

This doesn't allow getting new pypi packages, but see @dusktreader's comment [here](https://github.com/python-poetry/poetry/issues/4438#issuecomment-1082475509): > `$ poetry cache clear pypi:snick:1.2.0`

python-poetry / poetry

`poetry cache clear` should be able to clear all information for a single package #4438

Feature Request