[Windows] Installing a package from git is leading to "No such file or directory" error

[anetbnd]

Description

When I install a package from git on windows by doing poetry add git+https://github.com/BerriAI/litellm.git, I get an error:

[Errno 2] No such file or directory: b'<path to the python env>\\src\\litellm\\.git\\refs\\remotes\\origin\\dependabot\\npm_and_yarn\\docs\\my-website\\trim-and-docusaurus\\core-and-docusaurus\\plugin-google-gtag-and-docusaurus\\plugin-ideal-image-and-docusaurus\\preset-classic--removed.lock'

Where <path to the python env> is just a placeholder for a longer base path to my python environment. If I do the same on Mac, it works without issues.

What I think is, that the path is just too long for windows. Because Windows only supports path up to 260 characters. If you want to use paths longer than that you have to use the prefix \\?\, which allows path to up to 32.000 characters. See: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

If I run the command with -vvv, I get the following output:

Using virtualenv: <path to the python env>
Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[win32ctypes:win32ctypes.core.cffi] Loaded cffi backend
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Using keyring backend 'Windows WinVaultKeyring'
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3:urllib3.connectionpool] https://github.com:443 "GET /BerriAI/litellm.git/info/refs?service=git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/BerriAI/litellm.git at 'HEAD' to <path to the python env>\src\litellm

  Stack trace:

  20  <path to the python env>\Lib\site-packages\cleo\application.py:327 in run
       325│
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  19  <path to the python env>\Lib\site-packages\poetry\console\application.py:190 in _run
       188│         self._load_plugins(io)
       189│
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│

  18  <path to the python env>\Lib\site-packages\cleo\application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│

  17  <path to the python env>\Lib\site-packages\cleo\application.py:473 in _run_command
       471│
       472│         if error is not None:
     → 473│             raise error
       474│
       475│         return terminate_event.exit_code

  16  <path to the python env>\Lib\site-packages\cleo\application.py:457 in _run_command
       455│
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  15  <path to the python env>\Lib\site-packages\cleo\commands\base_command.py:117 in run
       115│         io.input.validate()
       116│
     → 117│         return self.execute(io) or 0
       118│
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:

  14  <path to the python env>\Lib\site-packages\cleo\commands\command.py:61 in execute
        59│
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1

  13  <path to the python env>\Lib\site-packages\poetry\console\commands\add.py:164 in handle
       162│             return 0
       163│
     → 164│         requirements = self._determine_requirements(
       165│             packages,
       166│             allow_prereleases=self.option("allow-prereleases"),

  12  <path to the python env>\Lib\site-packages\poetry\console\commands\init.py:375 in _determine_requirements
       373│
       374│         result = []
     → 375│         for requirement in self._parse_requirements(requires):
       376│             if "git" in requirement or "url" in requirement or "path" in requirement:
       377│                 result.append(requirement)

  11  <path to the python env>\Lib\site-packages\poetry\console\commands\init.py:441 in _parse_requirements
       439│             cwd=cwd,
       440│         )
     → 441│         return [parser.parse(requirement) for requirement in requirements]
       442│
       443│     def _format_requirements(self, requirements: list[dict[str, str]]) -> Requirements:

  10  <path to the python env>\Lib\site-packages\poetry\utils\dependency_specification.py:89 in parse
        87│
        88│         specification = (
     →  89│             self._parse_url(requirement)
        90│             or self._parse_path(requirement)
        91│             or self._parse_simple(requirement)

   9  <path to the python env>\Lib\site-packages\poetry\utils\dependency_specification.py:149 in _parse_url
       147│
       148│         if url_parsed.scheme in GIT_URL_SCHEMES:
     → 149│             return self._parse_git_url(requirement)
       150│
       151│         if url_parsed.scheme in ["http", "https"]:

   8  <path to the python env>\Lib\site-packages\poetry\utils\dependency_specification.py:133 in _parse_git_url
       131│
       132│         source_root = self._env.path.joinpath("src") if self._env else None
     → 133│         package = self._direct_origin.get_package_from_vcs(
       134│             "git",
       135│             url=url.url,

   7  <path to the python env>\llmaas\Lib\site-packages\poetry\packages\direct_origin.py:106 in get_package_from_vcs
       104│             raise ValueError(f"Unsupported VCS dependency {vcs}")
       105│
     → 106│         return _get_package_from_git(
       107│             url=url,
       108│             branch=branch,

   6  <path to the python env>\Lib\site-packages\poetry\packages\direct_origin.py:32 in _get_package_from_git
        30│     source_root: Path | None = None,
        31│ ) -> Package:
     →  32│     source = Git.clone(
        33│         url=url,
        34│         source_root=source_root,

   5  <path to the python env>\Lib\site-packages\poetry\vcs\git\backend.py:455 in clone
       453│         try:
       454│             if not cls.is_using_legacy_client():
     → 455│                 local = cls._clone(url=url, refspec=refspec, target=target)
       456│                 cls._clone_submodules(repo=local)
       457│                 return local

   4  <path to the python env>\Lib\site-packages\poetry\vcs\git\backend.py:288 in _clone
       286│             (b"refs/tags", b"refs/tags"),
       287│         }:
     → 288│             local.refs.import_refs(
       289│                 base=base,
       290│                 other={

   3  <path to the python env>\Lib\site-packages\dulwich\refs.py:189 in import_refs
        187│                 to_delete.add(name)
        188│             else:
     →  189│                 self.set_if_equals(
        190│                     b"/".join((base, name)), None, value, message=message
        191│                 )

   2  <path to the python env>\Lib\site-packages\dulwich\refs.py:897 in set_if_equals
        895│
        896│         ensure_dir_exists(os.path.dirname(filename))
     →  897│         with GitFile(filename, "wb") as f:
        898│             if old_ref is not None:
        899│                 try:

   1  <path to the python env>\Lib\site-packages\dulwich\file.py:93 in GitFile
        91│         raise OSError("text mode not supported for Git files")
        92│     if "w" in mode:
     →  93│         return _GitFile(filename, mode, bufsize, mask)
        94│     else:
        95│         return open(filename, mode, bufsize)

  FileNotFoundError

  [Errno 2] No such file or directory: b'<path to the python env>\\src\\litellm\\.git\\refs\\remotes\\origin\\dependabot\\npm_and_yarn\\docs\\my-website\\trim-and-docusaurus\\core-and-docusaurus\\plugin-google-gtag-and-docusaurus\\plugin-ideal-image-and-docusaurus\\preset-classic--removed.lock'

  at <path to the python env>\Lib\site-packages\dulwich\file.py:149 in __init__
      145│             self._lockfilename = self._filename + b".lock"
      146│         else:
      147│             self._lockfilename = self._filename + ".lock"
      148│         try:
    → 149│             fd = os.open(
      150│                 self._lockfilename,
      151│                 os.O_RDWR | os.O_CREAT | os.O_EXCL | getattr(os, "O_BINARY", 0),
      152│                 mask,
      153│             )

I think poetry must add the prefix for long windows paths to the source_root (or base path) where it wants the git clone operation to take part.

In my code, which needs to run on different systems, I often use a function like that:

    def adapt_for_long_windows_path_if_necessary(path: str) -> str:
        if os.name == "nt":  # pragma: no cover - system dependent
            path = '\\\\?\\' + str(Path(path).absolute().resolve())  # pragma: no cover - system dependent
        return path

Here it is necessary to make the path absolute, since the prefix only works with absolute paths.

Workarounds

I can install the package by pip directly and run operations like poetry lock on a linux client. But this is not a workaround for everyone.

Poetry Installation Method

pip

Operating System

Windows 11

Poetry Version

Poetry (version 1.8.3)

Poetry Configuration

cache-dir = "<a path in my user directory>"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
repositories.<private-repro>.url = "<url to private repro>"
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}\\virtualenvs"  # <a path to my user directory>
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

Platform: "win-amd64"
Python version: "3.12"
Current installation scheme: "nt"

Paths:
        data = "<path to the python env>"
        include = "<path to the python env>\Include"
        platinclude = "<path to the python env>\Include"
        platlib = "<path to the python env>\Lib\site-packages"
        platstdlib = "<path to the python env>\Lib"
        purelib = "<path to the python env>\Lib\site-packages"
        scripts = "<path to the python env>\Scripts"
        stdlib = "<path to the python env>\Lib"

Variables:
        BINDIR = "<path to the python env>"
        BINLIBDEST = "<path to the python env>\Lib"
        EXE = ".exe"
        EXT_SUFFIX = ".cp312-win_amd64.pyd"
        INCLUDEPY = "<path to the python env>\Include"
        LIBDEST = "<path to the python env>\Lib"
        TZPATH = "<path to the python env>\share\zoneinfo"
        VERSION = "312"
        VPATH = "..\.."
        abiflags = ""
        base = "<path to the python env>"
        exec_prefix = "<path to the python env>"
        installed_base = "<path to the python env>"
        installed_platbase = "<path to the python env>"
        platbase = "<path to the python env>"
        platlibdir = "DLLs"
        prefix = "<path to the python env>"
        projectbase = "<path to the python env>"
        py_version = "3.12.3"
        py_version_nodot = "312"
        py_version_nodot_plat = "312"
        py_version_short = "3.12"
        srcdir = "<path to the python env>"
        userbase = "<path to my user directory>\AppData\Roaming\Python"

Example pyproject.toml

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["me"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

see above

[radoering]

I do not think that we want to add special handling for long paths on Windows because it makes things complicated. For example, iirc you have to pay attention that both paths (or none) are prefixed with \\?\ when using relative_to.

If you need support for long paths, your best option is to set the registry key as described in Enable Long Paths in Windows 10, Version 1607, and Later.

[anetbnd]

Its my working computer, I don't have administrator rights and I don't have access to the registry.

This simply means, you don't want that certain users are able to use your software. There is no real workaround available, if you don't have access to a different system. Then please document clearly on the first page, that windows is not fully supported by poetry and there might be issues in certain cases when using windows.

[Xanderplayz16]

Its my working computer, I don't have administrator rights and I don't have access to the registry.

This simply means, you don't want that certain users are able to use your software. There is no real workaround available, if you don't have access to a different system. Then please document clearly on the first page, that windows is not fully supported by poetry and there might be issues in certain cases when using windows.

You could use symlinks then. Just mklink /d C:/Users/your_user/your_very_long_project_path C:/Users/your_user/shortcut_to_project. Now, just use the newly created symlink instead of the long project path. It isn't perfect, but it will work in your case.

[anetbnd]

Its my working computer, I don't have administrator rights and I don't have access to the registry. This simply means, you don't want that certain users are able to use your software. There is no real workaround available, if you don't have access to a different system. Then please document clearly on the first page, that windows is not fully supported by poetry and there might be issues in certain cases when using windows.

You could use symlinks then. Just mklink /d C:/Users/your_user/your_very_long_project_path C:/Users/your_user/shortcut_to_project. Now, just use the newly created symlink instead of the long project path. It isn't perfect, but it will work in your case.

Yeah, it would be a dirty workaround, messing up with the file system and installation paths. But actually we found a simpler solution: We substituted poetry with pdm, because pdm can handle long path names and is otherwise more or less the same as poetry.