Environment markers do not work when setting a dependencies `path`

[bmitc]

Description

Poetry does not respect a dependency which has an environment marker set and seemingly tries to resolve the other components when a marker dictates that the dependency is not valid for the given environment.

For example, on Windows 11, add a new dependency like the following:

[tool.poetry.dependencies]
python = "^3.12"
test = { path = "/only/exists/on/linux", markers = "sys_platform == 'linux'" }

Then when running any Poetry command, such as poetry install, Poetry fails to run and complains:

> poetry install
Path C:\only\exists\on\linux for test does not exist
Creating virtualenv test-markers in C:\Users\<user>\Desktop\test-markers\.venv
Updating dependencies
Resolving dependencies... (0.0s)

Path C:\only\exists\on\linux for test does not exist

This is a particular dependency which I don't have any control of on Linux and is installed differently on Windows. Thus, I set the marker to install the dependency via path only when on Linux. But now, Poetry fails to install anything.

Workarounds

None

Poetry Installation Method

pipx

Operating System

Windows 11

Poetry Version

Poetry (version 1.8.3)

Poetry Configuration

> poetry config --list
Path C:\only\exists\on\linux for test does not exist
cache-dir = "C:\\Users\\<user>\\AppData\\Local\\pypoetry\\Cache"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.ignore-conda-env = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}\\virtualenvs"  # C:\Users\bmitc\AppData\Local\pypoetry\Cache\virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

> python -m sysconfig
Platform: "win-amd64"
Python version: "3.12"
Current installation scheme: "nt"

Paths:
        data = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        include = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Include"
        platinclude = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Include"
        platlib = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Lib\site-packages"
        platstdlib = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Lib"
        purelib = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Lib\site-packages"
        scripts = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Scripts"
        stdlib = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Lib"

Variables:
        BINDIR = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        BINLIBDEST = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Lib"
        EXE = ".exe"
        EXT_SUFFIX = ".cp312-win_amd64.pyd"
        INCLUDEPY = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Include"
        LIBDEST = "C:\Users\<user>\AppData\Local\Programs\Python\Python312\Lib"
        TZPATH = ""
        VERSION = "312"
        VPATH = "..\.."
        abiflags = ""
        base = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        exec_prefix = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        installed_base = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        installed_platbase = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        platbase = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        platlibdir = "DLLs"
        prefix = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        projectbase = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        py_version = "3.12.1"
        py_version_nodot = "312"
        py_version_nodot_plat = "312"
        py_version_short = "3.12"
        srcdir = "C:\Users\<user>\AppData\Local\Programs\Python\Python312"
        userbase = "C:\Users\<user>\AppData\Roaming\Python"

Example pyproject.toml

[tool.poetry]
name = "test-markers"
version = "0.1.0"
description = ""
authors = ["user <user@users.noreply.github.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
test = { path = "/only/exists/on/linux", markers = "sys_platform == 'linux'" }

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

> poetry -vvv install
Path C:\only\exists\on\linux for test does not exist
Loading configuration file C:\Users\<user>\AppData\Roaming\pypoetry\config.toml
Using virtualenv: C:\Users\<user>\Desktop\test-markers\.venv
Updating dependencies
Resolving dependencies...
   1: fact: test-markers is 0.1.0
   1: derived: test-markers
   1: Version solving took 0.001 seconds.
   1: Tried 1 solutions.

  Stack trace:

  18  ~\AppData\Roaming\Python\Python312\site-packages\cleo\application.py:327 in run
       325│
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  17  ~\AppData\Roaming\Python\Python312\site-packages\poetry\console\application.py:190 in _run
       188│         self._load_plugins(io)
       189│
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│

  16  ~\AppData\Roaming\Python\Python312\site-packages\cleo\application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│

  15  ~\AppData\Roaming\Python\Python312\site-packages\cleo\application.py:473 in _run_command
       471│
       472│         if error is not None:
     → 473│             raise error
       474│
       475│         return terminate_event.exit_code

  14  ~\AppData\Roaming\Python\Python312\site-packages\cleo\application.py:457 in _run_command
       455│
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  13  ~\AppData\Roaming\Python\Python312\site-packages\cleo\commands\base_command.py:117 in run
       115│         io.input.validate()
       116│
     → 117│         return self.execute(io) or 0
       118│
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:

  12  ~\AppData\Roaming\Python\Python312\site-packages\cleo\commands\command.py:61 in execute
        59│
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1

  11  ~\AppData\Roaming\Python\Python312\site-packages\poetry\console\commands\install.py:153 in handle
       151│         self.installer.verbose(self.io.is_verbose())
       152│
     → 153│         return_code = self.installer.run()
       154│
       155│         if return_code != 0:

  10  ~\AppData\Roaming\Python\Python312\site-packages\poetry\installation\installer.py:104 in run
       102│             self.verbose(True)
       103│
     → 104│         return self._do_install()
       105│
       106│     def dry_run(self, dry_run: bool = True) -> Installer:

   9  ~\AppData\Roaming\Python\Python312\site-packages\poetry\installation\installer.py:241 in _do_install
       239│                 source_root=self._env.path.joinpath("src")
       240│             ):
     → 241│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       242│         else:
       243│             self._io.write_line("Installing dependencies from lock file")

   8  ~\AppData\Roaming\Python\Python312\site-packages\poetry\puzzle\solver.py:71 in solve
        69│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        70│             start = time.time()
     →  71│             packages, depths = self._solve()
        72│             end = time.time()
        73│

   7  ~\AppData\Roaming\Python\Python312\site-packages\poetry\puzzle\solver.py:154 in _solve
       152│ 
       153│         try:
     → 154│             result = resolve_version(self._package, self._provider)
       155│
       156│             packages = result.packages

   6  ~\AppData\Roaming\Python\Python312\site-packages\poetry\mixology\__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│
     →  18│     return solver.solve()
        19│

   5  ~\AppData\Roaming\Python\Python312\site-packages\poetry\mixology\version_solver.py:175 in solve
       173│             while next is not None:
       174│                 self._propagate(next)
     → 175│                 next = self._choose_package_version()
       176│
       177│             return self._result()

   4  ~\AppData\Roaming\Python\Python312\site-packages\poetry\mixology\version_solver.py:514 in _choose_package_version
       512│             package = locked
       513│
     → 514│         package = self._provider.complete_package(package)
       515│
       516│         conflict = False

   3  ~\AppData\Roaming\Python\Python312\site-packages\poetry\puzzle\provider.py:568 in complete_package
       566│                     if locked is not None and locked.package.is_same_package_as(dep):
       567│                         continue
     → 568│                     self.search_for_direct_origin_dependency(dep)
       569│
       570│         dependencies = self._get_dependencies_with_overrides(_dependencies, package)

   2  ~\AppData\Roaming\Python\Python312\site-packages\poetry\puzzle\provider.py:253 in search_for_direct_origin_dependency
       251│         elif dependency.is_directory():
       252│             dependency = cast("DirectoryDependency", dependency)
     → 253│             package = self._search_for_directory(dependency)
       254│
       255│         elif dependency.is_url():

   1  ~\AppData\Roaming\Python\Python312\site-packages\poetry\puzzle\provider.py:359 in _search_for_directory
       357│ 
       358│     def _search_for_directory(self, dependency: DirectoryDependency) -> Package:
     → 359│         dependency.validate(raise_error=True)
       360│         package = self._direct_origin.get_package_from_directory(dependency.full_path)
       361│

  ValueError

  Path C:\only\exists\on\linux for test does not exist

  at ~\AppData\Roaming\Python\Python312\site-packages\poetry\core\packages\path_dependency.py:82 in validate
       78│     def validate(self, *, raise_error: bool) -> bool:
       79│         if not self._validation_error:
       80│             return True
       81│         if raise_error:
    →  82│             raise ValueError(self._validation_error)
       83│         logger.warning(self._validation_error)
       84│         return False
       85│
       86│     @property

[dimbleby]

Resolution still has to happen (how else should the lock file be created?)

Please close

[bmitc]

Why does resolution have to happen when the dependency does not apply to the given platform?

What is the point of environment markers then?

[bmitc]

Also, if you're not a maintainer, then please don't suggest to close this.

This is an issue and a scenario that Poetry needs to handle, whether it's considered a bug or a feature request.

[dimbleby]

Why does resolution have to happen when the dependency does not apply to the given platform?

we already did this. How else should the lock file be created?

You can lock on a platform where the path is available and then install using that lockfile. Or you can just pip install all along

This should be closed because it is working correctly, and as intended.

[bmitc]

How else should the lock file be created?

Please elaborate. What is the lock file doing in this scenario?

You can lock on a platform where the path is available and then install using that lockfile.

I had in fact already created the lock file on Linux where that path existed. This issue was discovered when I went to add the Windows-specific dependency.

[dimbleby]

poetry builds a cross-platform lockfile to make sure that your requirements are consistent, and that your installation is reproducible. When doing that the platform on which you are executing the command is more or less irrelevant: it is a cross-platform lockfile so poetry has to consider requirements for all platforms.

So if your lockfile is out of date - eg because you are adding a new requirement - then poetry must rebuild it, and to do that it must be able to find all dependencies.

There is no bug here.

[bmitc]

You keep referring to implementation details in order to reject a high-level use case.

In this scenario, it appears that the path and markers dependency keys are incompatible. This should either work as is and is thus a bug, is a feature request, or it is a documentation issue.

You stating that this isn't a bug over and over isn't contributing anything or helpful.

[bmitc]

And it is not clear as to why Poetry needs to rebuild or re-resolve the path and markers specified dependency. Its lock component has no reason to be updated.

[dimbleby]

You stating that this isn't a bug over and over isn't contributing anything or helpful.

Nevertheless: it isn't! And will surely be closed as such sooner or later.

Documentation contributions are usually relatively straightforward and welcome. If you would like to see change happen then the most likely way to do that would be to submit a merge request clarifying whatever part of the docs you looked at and which did not help you to understand poetry behaviour.

[bmitc]

Documentation contributions are usually relatively straightforward and welcome. If you would like to see change happen then the most likely way to do that would be to submit a merge request clarifying whatever part of the docs you looked at and which did not help you to understand poetry behaviour.

This is obvious. I would be happy to contribute a documentation change, but the point of this is to discuss the nuances of this issue. Even if it happens to be a documentation issue, which information hasn't been presented to convince of that, then this issue should remain open until the documentation is updated.

[dimbleby]

If you would like to contribute improved documentation then you can just go ahead and do it, no need for an issue too.

However it is true that those with the power to tidy up do seem content to allow the pile of unloved issues to accumulate...

Hopefully my last but one is most of the content you'd need.

[radoering]

I will close this issue because as already mentioned it is not a bug. https://github.com/python-poetry/poetry/issues/9679#issuecomment-2340523397 explains why. However, you cannot create a universal lock file without having access to all dependencies. You can poetry install with an existing up-to-date lock file without having access to dependencies that are not required on the target platform.

I did not find anything in the docs where this is explained in detail so maybe we should add an FAQ entry or put it somewhere else. If you do not feel comfortable to contribute to the docs, feel free to open a documentation issue. However, I do not think it makes sense to keep this issue open because it is not a bug.

[bmitc]

So, to be clear: it is not possible to have a path specified dependency that is platform dependent in Poetry. Is that correct?

[Secrus]

It is. But it doesn't mean it can be absent when locking on another platform

[bmitc]

It is correct, or it is possible to have a path dependency that is platform dependent? If you mean the latter, based upon your comment that "it doesn't mean it can be absent when locking on another platform", then that's a very constrained use case. Basically, it only applies when relative paths are used. Even if Poetry resolves paths in a cross-platform independent way, it is very unlikely for a path dependency on Windows to be in the same location (i.e, have the same path) as on Linux.