Poetry self update fails on Windows with Access Denied, leaving poetry in a broken state

[iTitus]

• Poetry version: Current version is 1.4.0, but it has happened to me multiple times when updating poetry <!-- Replace with version, e.g. from poetry --version -->
• Python version: System Python is 3.10.5
• OS version and name: Windows 10
• pyproject.toml: N/A

• [x] I am on the latest stable Poetry version, installed using a recommended method. (not really applicable)
• [x] I have searched the issues of this repo and believe that this is not a duplicate. (there are similar issues, but none have a solution/activity)
• [x] I have consulted the FAQ and blog for any relevant entries or release notes.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below. (output included below, but not verbose)

Issue

This has happened to me for multiple version updates in a row now. Whenever upgrading poetry with poetry self update pip fails with an OSError and leaves Poetry in a broken state. A reinstall fixes that.

There are two problems here:

1. Pip crashes due to an OSError: this does not happen when normally running pip or using poetry to install/update venvs, only when upgrading poetry itself
2. Poetry does not rollback any changes after pip crashes, thus leaving itself in a broken and unusable state

poetry self update
Updating Poetry version ...

Using version ^1.4.0 for poetry

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 4 installs, 4 updates, 0 removals

  • Installing colorama (0.4.6)
  • Updating poetry-core (1.4.0 -> 1.5.1)
  • Installing pyproject-hooks (1.0.0)
  • Installing build (0.10.0)
  • Updating dulwich (0.20.50 -> 0.21.3)
  • Updating poetry-plugin-export (1.2.0 -> 1.3.0)
  • Installing installer (0.6.0)

  CalledProcessError

  Command 'C:\Users\iTitus\AppData\Roaming\pypoetry\venv\Scripts\python.exe -m pip install --disable-pip-version-check --isolated --no-input --prefix C:\Users\iTitus\AppData\Roaming\pypoetry\venv --upgrade --no-deps C:\Users\iTitus\AppData\Local\pypoetry\Cache\artifacts\88\74\c6\b2de65e4cd5082ed96394c6c1ca241ad7839b76a5b9cc35af5c0b84ecb\dulwich-0.21.3-cp311-cp311-win_amd64.whl' returned non-zero exit status 1.

  at C:\Program Files\Python311\Lib\subprocess.py:569 in run
       565│             # We don't call process.wait() as .__exit__ does that for us.
       566│             raise
       567│         retcode = process.poll()
       568│         if check and retcode:
    →  569│             raise CalledProcessError(retcode, process.args,
       570│                                      output=stdout, stderr=stderr)
       571│     return CompletedProcess(process.args, retcode, stdout, stderr)
       572│
       573│

The following error occurred when trying to handle this error:

  EnvCommandError

  Command C:\Users\iTitus\AppData\Roaming\pypoetry\venv\Scripts\python.exe -m pip install --disable-pip-version-check --isolated --no-input --prefix C:\Users\iTitus\AppData\Roaming\pypoetry\venv --upgrade --no-deps C:\Users\iTitus\AppData\Local\pypoetry\Cache\artifacts\88\74\c6\b2de65e4cd5082ed96394c6c1ca241ad7839b76a5b9cc35af5c0b84ecb\dulwich-0.21.3-cp311-cp311-win_amd64.whl errored with the following return code 1, and output:
  Processing c:\users\ititus\appdata\local\pypoetry\cache\artifacts\88\74\c6\b2de65e4cd5082ed96394c6c1ca241ad7839b76a5b9cc35af5c0b84ecb\dulwich-0.21.3-cp311-cp311-win_amd64.whl
  Installing collected packages: dulwich
    Attempting uninstall: dulwich
      Found existing installation: dulwich 0.20.50
      Uninstalling dulwich-0.20.50:
        Successfully uninstalled dulwich-0.20.50
  ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'C:\\Users\\iTitus\\AppData\\Roaming\\pypoetry\\venv\\Lib\\site-packages\\~ulwich\\_diff_tree.cp311-win_amd64.pyd'
  Check the permissions.

  at venv\Lib\site-packages\poetry\utils\env.py:1540 in _run
      1536│                 output = subprocess.check_output(
      1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1538│                 )
      1539│         except CalledProcessError as e:
    → 1540│             raise EnvCommandError(e, input=input_)
      1541│
      1542│         return decode(output)
      1543│
      1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:

  PoetryException

  Failed to install C:/Users/iTitus/AppData/Local/pypoetry/Cache/artifacts/88/74/c6/b2de65e4cd5082ed96394c6c1ca241ad7839b76a5b9cc35af5c0b84ecb/dulwich-0.21.3-cp311-cp311-win_amd64.whl

  at venv\Lib\site-packages\poetry\utils\pip.py:58 in pip_install
       54│
       55│     try:
       56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
    →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       59│

[lfrybouriaux]

Not sure if this helps, but I was upgrading from 1.2.2 to 1.4.0 today using this method and I encountered the same kind of OSError on Windows 10, albeit on a different file, amongst other issues with dependencies raising assertions. Note I ran poetry self update in an anaconda powershell. I then checked the local environment variables and found that I didn't have the expected [user]\AppData\Roaming\Python\Scripts\ in my PATH anymore (pretty sure I had it before), but [user]\.local\bin and it was using old poetry from there. So I replaced with the new env dir in my PATH after reinstalling manually in a system powershell. Now it seems to work ok. Doing this didn't delete my envs.

[iTitus]

Based on the logged commands it seems like the correct pip is executed.

[lfrybouriaux]

I'm guessing poetry uses the pip that comes with the python that was used to install it?

In a nutshell, this issue was solved for me by reinstalling poetry in system powershell, with a system python (3.9.13 in my case), completely outside of anaconda, and double checking what the installer says about poetry path with what is in my actual PATH on windows.

[iTitus]

The recent update gave me this (different) error:

> poetry self update
Updating Poetry version ...

Using version ^1.4.1 for poetry

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 0 installs, 3 updates, 0 removals

  • Updating poetry-core (1.5.1 -> 1.5.2)
  • Updating installer (0.6.0 -> 0.7.0)
  • Updating poetry (1.4.0 -> 1.4.1)

  CalledProcessError

  Command '['C:\\Users\\iTitus\\AppData\\Roaming\\pypoetry\\venv\\Scripts\\python.exe', '-m', 'pip', 'uninstall', 'poetry', '-y']' returned non-zero exit status 2.

  at C:\Program Files\Python311\Lib\subprocess.py:571 in run
       567│             # We don't call process.wait() as .__exit__ does that for us.
       568│             raise
       569│         retcode = process.poll()
       570│         if check and retcode:
    →  571│             raise CalledProcessError(retcode, process.args,
       572│                                      output=stdout, stderr=stderr)
       573│     return CompletedProcess(process.args, retcode, stdout, stderr)
       574│
       575│

The following error occurred when trying to handle this error:

  EnvCommandError

  Command ['C:\\Users\\iTitus\\AppData\\Roaming\\pypoetry\\venv\\Scripts\\python.exe', '-m', 'pip', 'uninstall', 'poetry', '-y'] errored with the following return code 2

  Output:
  Found existing installation: poetry 1.4.0
  Uninstalling poetry-1.4.0:
  ERROR: Exception:
  Traceback (most recent call last):
    File "C:\Program Files\Python311\Lib\shutil.py", line 825, in move
      os.rename(src, real_dst)
  PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'c:\\users\\ititus\\appdata\\roaming\\pypoetry\\venv\\scripts\\poetry.exe' -> 'C:\\Users\\iTitus\\AppData\\Local\\Temp\\pip-uninstall-ux9bwfj8\\poetry.exe'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "C:\Users\iTitus\AppData\Roaming\pypoetry\venv\Lib\site-packages\pip\_internal\cli\base_command.py", line 160, in exc_logging_wrapper
      status = run_func(*args)
               ^^^^^^^^^^^^^^^
    File "C:\Users\iTitus\AppData\Roaming\pypoetry\venv\Lib\site-packages\pip\_internal\commands\uninstall.py", line 105, in run
      uninstall_pathset = req.uninstall(
                          ^^^^^^^^^^^^^^
    File "C:\Users\iTitus\AppData\Roaming\pypoetry\venv\Lib\site-packages\pip\_internal\req\req_install.py", line 664, in uninstall
      uninstalled_pathset.remove(auto_confirm, verbose)
    File "C:\Users\iTitus\AppData\Roaming\pypoetry\venv\Lib\site-packages\pip\_internal\req\req_uninstall.py", line 373, in remove
      moved.stash(path)
    File "C:\Users\iTitus\AppData\Roaming\pypoetry\venv\Lib\site-packages\pip\_internal\req\req_uninstall.py", line 271, in stash
      renames(path, new_path)
    File "C:\Users\iTitus\AppData\Roaming\pypoetry\venv\Lib\site-packages\pip\_internal\utils\misc.py", line 312, in renames
      shutil.move(old, new)
    File "C:\Program Files\Python311\Lib\shutil.py", line 846, in move
      os.unlink(src)
  PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'c:\\users\\ititus\\appdata\\roaming\\pypoetry\\venv\\scripts\\poetry.exe'

  at venv\Lib\site-packages\poetry\utils\env.py:1545 in _run

> poetry self update
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\iTitus\AppData\Roaming\Python\Scripts\poetry.exe\__main__.py", line 4, in <module>
ModuleNotFoundError: No module named 'poetry.console'

[iTitus]

reinstalling works fine:

> (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

C:\Users\iTitus\AppData\Roaming\Python\Scripts

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.4.1)
Installing Poetry (1.4.1): Saving existing environment
Installing Poetry (1.4.1): Creating environment
Installing Poetry (1.4.1): Installing Poetry
Installing Poetry (1.4.1): Creating script
Installing Poetry (1.4.1): Done

Poetry (1.4.1) is installed now. Great!

To get started you need Poetry's bin directory (C:\Users\iTitus\AppData\Roaming\Python\Scripts) in your `PATH`
environment variable.

Alternatively, you can call Poetry explicitly with `C:\Users\iTitus\AppData\Roaming\Python\Scripts\poetry`.

You can test that everything is set up by executing:

`poetry --version`

> poetry --version
Poetry (version 1.4.1)

[finswimmer]

Honestly I have no idea how to proceed with this kind of issue :(

It is reported regularly, but it is not reliable reproducible. The only thing the reports seems to have in common is, that this happens under Windows and a antivirus software is running.

So if anyone has clue how to improve the situation ...

[dimbleby]

I don't honestly understand how this is expected to work - isn't using poetry to manage its own environment all but certain sometimes to remove packages that poetry is actively using?

A fairly regular error on this tracker is folk who have installed poetry in the project under management and then are surprised when adding and removing packages breaks that installation. I feel I must have missed something that makes it all fine, but isn't poetry self built around that same error?!?

[finswimmer]

isn't using poetry to manage its own environment all but certain sometimes to remove packages that poetry is actively using?

Good point, might be.

But I saw this problem several times in normal project environments as well, when adding/updating/removing packages there.

[dimbleby]

more or less by definition: not the same problem if it's not poetry self related. Perhaps you're thinking of #1031 etc (where https://github.com/python-poetry/poetry-core/pull/460 looks close to happening)

IMO if no-one can see a reason that poetry self is not just broken then it should be deprecated and removed, tell folk to prefer eg pipx. Of course that's a non-trivial step and will take some buy-in...

[Leem0sh]

after using poetry self update

This happened to me as well for the second time with different version. Since Poetry command is not working now(I have an idea what to do now but) have anyone thought of putting some errors into FAQ for ppl who are not that experienced with Poetry when this happens? Honestly this being first update experience with poetry, I'd leave for good.

edit: Deleted everything from pypoetry in Roaming and Local. Reinstalled.

I noticed I had a process running in the background with python under poetry. Might be connected with the issue. Even tho now I tried to have processes running while downgrading/upgrading but the issue didn't replicate. Might go tweak a little with the issue and report if anything happens.

[adam-grant-hendry]

IMO if no-one can see a reason that poetry self is not just broken then it should be deprecated and removed, tell folk to prefer eg pipx. Of course that's a non-trivial step and will take some buy-in...

We should state WHY pipx works when poetry commands don't. To be clear, The [WinError 5] permission errors have historically happened with more than just the poetry self commands. Also not sure if poetry self is breaking for non-Windows users, so I'm generally against changing the code base just for this (also since we'd be changing the traditional way plugins are added).

List of Issues

poetry add [WinError 5] Errors

• 2523

• 4346

poetry install [WinError 5] Errors

• 8047

[dimbleby]

those issues are only superficially similar: all of them are related to a quite different problem (where python fails to clean up temporary directories on Windows, start at https://github.com/python/cpython/pull/24793 if you want to dig into it). per an earlier comment: if it's not poetry self, then more or less by definition it's not this.

this issue is about: the poetry (python) process cannot update files while it is itself using them - or anyway, not without things going wrong.

plainly pipx updating poetry's files is fine

[adam-grant-hendry]

I'm still against removing poetry self

[iTitus]

But can there be a fix for the issue where poetry literally breaks itself when this exception is thrown? Catch it and roll-back maybe?

[adam-grant-hendry]

The recent update gave me this (different) error:

@iTitus I personally experience this error because when poetry is installed, added, or updated:

• Updating poetry-core (1.5.1 -> 1.5.2) • Updating poetry (1.4.0 -> 1.4.1)

pip installs ’poetrys entrypoint script (poetry.exe)

[tool.poetry.scripts]
poetry = "poetry.console.application:main"

to venv/Scripts. Hence you’re running poetry.exe to update poetry while trying to then overwrite poetry.

This is similar to the problem that happens when people use pip to update pip:

pip update pip

and is why

python -m pip install —update pip

works (or alternatively py -m if you prefer the Windows Python Launcher). (See Bret Cannon’s article for reference).

@dimbleby @finswimmer I believe this to be the culprit of the WinErrors.

[dimbleby]

some but not all of the reports in this thread are about updating poetry.exe - https://github.com/python-poetry/poetry/issues/7610#issuecomment-1483025421 is, the opener is not

but in general: yes, this is what I have been saying throughout - the problem is poetry trying to update files while it is using them. poetry.exe is indeed one such file.

[adam-grant-hendry]

but in general: yes, this is what I have been saying throughout

Wholeheartedly agree; apologies if it took me a while to get there.

This is also why using ‘pipx’ works, as you have rightly been pointing out.

[adam-grant-hendry]

Out of curiosity, if one pip installs poetry, would the incantation

py -m poetry …

also fix this? (Here py being the Python Launcher for Windows)

[Tylersuard]

1. Open command prompt as administrator
2. If you created a virtual environment using venv, deactivate it

[ghost]

FWIW, programs which update themselves (including the executable which performs the update) can work around Windows' restrictions by renaming the executable instead of deleting it. Windows complains if you delete an executable which is being run, but it does not complain if the executable is renamed (as long as it remains on the same drive, I think). The renamed file can be removed by the new executable the next time it is run, if desired.

So, a way for poetry self update to work on Windows, is to have it rename poetry.exe to poetry.exe.old (or something like that) before doing the rest of the upgrade. Obviously this feels incredibly dirty but it would improve the UX. You would also need to handle the update being aborted mid-way through so that you can rename poetry.exe.old back to poetry.exe.

[dimbleby]

none of the reported errors are about the poetry executable, the above is not relevant

[ghost]

To be fair, this error was to do with poetry.exe: https://github.com/python-poetry/poetry/issues/7610#issuecomment-1483025421

And the issue with the Poetry executable is effectively the same cause as the issue with other permission errors reported here, I would argue it's still relevant.

But you're right - the OSErrors are also arising from Poetry's dependencies being updated whilst Poetry is using them, especially DLLs (.pyd files) such as Dulwich. However, if a poetry self update happens where none of the in-use DLLs are updated, you will always have a permission error with poetry.exe - that one is guaranteed. The EXE renaming hack won't prevent the DLL update errors. And I'm assuming it's impossible for Poetry to prevent those DLLs from being loaded during the self update process.

Perhaps poetry self update should do nothing on Windows, and instead print an error message saying to update using the official installer again (if it was originally installed with the official installer)? It could print the command (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -.

[dimbleby]

Apparently I knew that this was sometimes about poetry.exe at https://github.com/python-poetry/poetry/issues/7610#issuecomment-1677595387...

But yes: it often isn‘t, so the renaming trick cannot be sufficient.

While this brokenness mostly reveals itself on Windows I strongly suspect that it's there, ready to strike, everywhere. So I still favour deprecating the whole thing.

(Another reason for wanting to lose poetry self update is that it isn't even expected to work for pipx-installed poetry - which is now the first recommended installation method. eg #7170, #8618. It really is becoming a footgun for those users who are unfortunate enough to try it.)

[ghost]

Interesting, I understand why you want to remove poetry self update since it appears to be a nexus for update issues. Then you would need to be clearer about how to update Poetry when installed using the official install.python-poetry.org script.

While this brokenness mostly reveals itself on Windows I strongly suspect that it's there, ready to strike, everywhere. So I still favour deprecating the whole thing.

Deleting/replacing in-use files on *nix systems is generally fine - when a file like a shared library is deleted (unlinked), the file record is removed from the directory (so no other programs can open it any more), but file is not deleted from the disk until all of the file's open handles are closed. So for the most part, I would say that applications which update themselves in-place work fine on *nix systems.

[finswimmer]

The recommended installation method is now pipx, which circumvent the problems described initially in this issue.