python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.8k stars 2.28k forks source link

Support for mutually exclusive dependencies across groups (e.g. path dependencies during development only) #1168

Open hozn opened 5 years ago

hozn commented 5 years ago

Question

I believe this issue is related to #668 -- or some of the comments therein -- but I believe my workflow / use case is also a little different from the origin of that issue.

Here's the background / what I'm trying to accomplish:

Here's how I would do this with setuptools / pip:

This all works reasonably well, though the initial environment setup is a pain. I'm really hoping to move to Poetry but I also feel that I "need" an equivalent to the above.

I can use "path dependencies" to point to my local shared libraries, but then these aren't going to work when it actually goes to build on the CI server (or on anyone else's workstation if they organize their files differently):

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }
# THIS ONLY WORKS ON MY WORKSTATION

If I specify the same dependency in [tool.poetry.dependencies] and [tool.poetry.dev-dependencies], it seems to always pick the one from the non-dev deps:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^4.1"
my-lib2 = "^1.12.3"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }

# IT ALWAYS TRIES TO FIND MATCHING VERSIONS ON THE PYPI SERVER, DOESN'T USE THE PATHS

But even if this did work, ideally the solution wouldn't involve me using path dependencies at all, since (in this case) these are specific to how my workstation is setup -- a coworker may organize things differently.

So, is there a current solution to solving this problem? Perhaps I've missed something in reading the docs. Thanks for any help!

KholdStare commented 2 years ago

Just wanted to chime in here to say this is really important. Currently setting up a monorepo with python services/libraries, and need to work around these current limitations. Trying out a set up similar to what @dermidgen proposed (https://github.com/dermidgen/python-monorepo/), but it is definitely a workaround, and doesn't work for all cases.

At the end of the day, poetry supports local paths for dependencies so I think there needs to be a sane outcome when running poetry build in this case. IMHO, it should most likely package all the local dependencies up into the built .whl file - that's the simplest solution. If something is local, and we're building a module to be installed somewhere, it makes sense to bundle any non-external dependencies with the build, because there is no other way to get them.

The approaches I am pursuing now involve makefiles, parsing the toml to get dependencies (and transitive dependencies, etc.) that are local, and making sure I have *.whl files for all of them, so they can be installed in the right order in a Docker image. This really needs to be handled by build tooling - i.e. poetry.

jonapich commented 2 years ago

@KholdStare I have built a solution around this problem (obtaining wheels from lock files + local dependencies) in coveo-stew.

poocat commented 2 years ago

Thought I might chime in, as I also would love to see support for optionally using local path dependencies, specifically for a "monorepo" use case.

Expanding on the "overrides file" idea from @NeilGirdhar, I feel like there is some precedent for this in the form of pip's constraints.

The same way that pip constraints files use a "subset" of the contents and syntax of requirements.txt files, a constraints file for poetry would use a "subset" of the context and syntax of pyproject.toml files. For instance:

if your pyproject.toml looks like:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^4.1"  # from your private package index, or wherever else this is published

then your constraints.toml might would look like:

[tool.poetry.dependencies]
my-lib1 = { path = "../lib1/", develop = true }

then, whenever you wanted to work on your app/library locally, you could configure your environment with something like

poetry install --constraints constraints.toml
mindstorm38 commented 2 years ago

I'm wondering why nobody mentioned the simpler approach of Cargo (rust), where path dependencies are converted to version dependencies when building/publishing (freezing the version of the path dependency at the time of the build). This would also avoid specifying/updating the required versions on every update. Maybe I've missed something.

We can also be more precise on constraints by changing the meaning of version when using path dependencies. I've a really early idea (and probably broken), for example here if the local dependency has version 1.2.3:

# Simple constraints:
dep = { path = "<path>", version = ">=x" }    # actual constraint: >=1
dep = { path = "<path>", version = ">=x.x" }  # actual constraint: >=1.2
dep = { path = "<path>", version = "^x.x" }   # actual constraint: ^1.2
dep = { path = "<path>", version = "^x.x, !=1.2.0" }   # we can also use classic constraints
# By default, this constraints can be used to allow only patch updates (according to sementic versionning):
dep = { path = "<path>", version = "~x.x.x" }                 # actual constraint: ~1.2.3
dep = { path = "<path>", version = ">=x.x.x, <x.{x+1}.0" }    # equivalent to the previous one, less readable, and {x+1} syntax is ugly imo

What do you think about this idea? I can try to work on a proof-of-concept pull request if you find that useful.

NeilGirdhar commented 2 years ago

@mindstorm38 In your example, are you storing the path dependencies in the shared repository file "pyproject.toml", which is visible to all users and developers? The point of local path dependencies is for an individual developer to override the settings in this file.

(Incidentally, since I'm commenting anyway, the nice thing about @poocat's idea of a "constraints.toml" is that you can have various sets of local override sets. Nice idea.)

mindstorm38 commented 2 years ago

@NeilGirdhar My idea is inspired from a personal project where the local path is only valid within the repository (it's a subproject structure, with dependencies between subprojects). It's also used like this with Cargo workspaces we can also use git submodules (even if I don't like such dependency management, I also know that a workspace issue is being discussed, but for me, it's not directly related to the current one and is not required for it). For me, dependencies within the pyproject.toml should not point outside the project's root, maybe overrides.toml is more suitable for what @poocat suggested, and can be used as a user-defined overrides to the pyproject.toml, but it seems off-topic for this issue, or I'm wrong about the issue?

NeilGirdhar commented 2 years ago

it's a subproject structure, with dependencies between subprojects

Maybe I've misunderstood, but I don't think that makes sense for a project that is uploaded to PyPI.

For me, dependencies within the pyproject.toml should not point outside the project's root

You mean that dependencies like this shouldn't exist?

mindstorm38 commented 2 years ago

Maybe I've misunderstood, but I don't think that makes sense for a project that is uploaded to PyPI.

In my case, I'm working with a single git repository, with a core python package and many optional add-ons packages, that I want to be distributed as well, but separately from the core package and dependent to it. It's like poetry and poetry-core, but within a single repo (it would be too complicated for my little project to make multiple repos, and many devs are in this case I guess).

You mean that dependencies like this shouldn't exist?

No, of course, I meant path-dependencies.

NeilGirdhar commented 2 years ago

In my case, I'm working with a single git repository,

Right, but poetry needs to support projects that aren't "monorepos", and also projects that are uploaded to PyPI.

No, of course, I meant path-dependencies.

The path dependencies that I and poobar are talking about do point outside the repository. This allows a Poetry "editable mode" akin to pip install -e.

mindstorm38 commented 2 years ago

Ok, I see. Just to clarify, my idea would allow at the same time monorepos, multi-repos and publishing to PyPI, that's the main goal of it, because for now poetry lack of monorepos support imo.

Do you think that this idea could fit into poetry, or contributing is a lost of time? I like monorepos, but this might be the wrong issue to discuss it, and in my case it's quite specific to path dependencies...

NeilGirdhar commented 2 years ago

I'm not sure if this is the right issue. If you look at the very first comment, the dependencies listed are:

my-lib1 = { path = "../lib1/", develop = true }

So it looks like this person wants to depend on things outside the repository in a kind of development ("editable") mode.

As for monorepos, I don't know much about them, but maybe start a new issue explaining what you want to poetry to do, and link this issue as related?

mindstorm38 commented 2 years ago

This is a good idea, thanks I'll work on that!

sinoroc commented 2 years ago

@mindstorm38 See #2270 maybe.

mindstorm38 commented 2 years ago

I know this one, but I disagree with this approach of a single pyproject.toml for a whole repo. But that's not the point because they don't talk about path dependencies and how to resolve them at build time.

jonapich commented 2 years ago

https://github.com/python-poetry/poetry/discussions/3646 is the discussion going about this.

joaoe commented 2 years ago

Hi.

My use case is simpler than the OPs.

In my pyproject.toml I have

[tool.poetry]
name = "lettuce"
version = "0.0.0"
description = "Salad is good for you"

[tool.poetry.dev-dependencies]
t-lettuce = {path = "tests", develop = true}

So during a regular non-dev build, the lettuce shall always be installed in my environment. During a dev install, I want my unit test module to also be installed as a python module, such that individual test files can have a from t_lettuce import helper_function. So far, this pyprojet.toml has worked for me, plus a setup.py file in the tests folder.

However, I'd like there not to be a requirement to put a setup.py or pyproject.toml inside the tests folder, since these are just stub files to fool poetry install.

NeilGirdhar commented 2 years ago

@joaoe Out of curiosity, why are your tests an entire package? Why not use pytest?

TBBle commented 2 years ago

@joaoe I think you might be asking for something that's not feasible in Python packaging. By my understanding, the set of "what packages are in the project dist" cannot be affected by install-time options like 'extras', as the included package list is defined at sdist/wheel creation time, not installation time.

A close match would be a feature request for the format field of the packages list to support another value, e.g. editable (adding to current sdist and wheel), which then would include that package only during an editable install (which is not the same as a 'dev' install, but might work for your use-case). The idea being that something like this would work:

[tool.poetry]
name = "lettuce"
version = "0.0.0"
description = "Salad is good for you"
packages = [
    { include = "lettuce", from = "src" },
    { include = "t-lettuce", from = "tests", format = "editable" },
]

and then when installed in an editable install, e.g. as the root package in a Poetry venv, or when installing as a pathed develop dependency from some other package; t_lettuce would exist as a module, but would not be included in either an sdist or a wheel built from the same source.

That would be a Poetry-specific behaviour, and might depend on how the Poetry implementation of PEP-660 functions: Poetry-core supports PEP-660 already, but Poetry itself still manages editable installs directly, not using poetry-core's PEP-660 support, so such a feature might need to be implemented in two places.

joaoe commented 2 years ago

Out of curiosity, why are your tests an entire package? Why not use pytest?

I'm using pytest. Want we want is simply for the tests/ folder to be populated in sys.path (without having to do that explicitly in code) so any test file can import stuff from other test files, and in a way that works for IDEs and tools to provide auto complete and other kinds of checks. Plus, I see other scenarios where I could have an extra folder with scripts and utilities which shall not be distributed, but still in the path when the project's virtualenv is active.

A close match would be a feature request for the format field of the packages list to support another value, e.g. editable (adding to current sdist and wheel), which then would include that package only during an editable install (which is not the same as a 'dev' install, but might work for your use-case).

That seems like a decent suggestion. But now that I think about it, tagging a package folder as a dev dependency would somehow allow for the feature of having say "production" and "development" builds of a package, where the "development" could include stuff like unit tests. Or is that already possible ?

AKuederle commented 2 years ago

@joaoe Having IDE support for import and auto completion for files within the tests/ folder should "just" work. The only thing you need to do is to make your tests forlder a proper python package by including __init__.py files in all folders. With that PyCharm and VSCode (just tested) allow for full auto completion and imports from one file in the test folder to another using either relative or absolute import path (import config from .base_test_file or import config from tests.base_test_file. No further configuration required.

So I think your usecase does not need any support from Poetry

TBBle commented 2 years ago

But now that I think about it, tagging a package folder as a dev dependency would someone allow for the feature of having say "production" and "development" builds of a package, where the "development" could include stuff like unit tests.

This is the thing that the Python packaging ecosystem doesn't handle in any way. There's no "variations" of a project in this sense, they would be two separate projects. The closest is the 'extras' mechanism which only changes dependencies at install-time, so you could have your test helpers (and tests, I guess...) in a separate package that your package depends on with a 'dev' extra.

However, then you're back to the core issue of this ticket, which is that Poetry can't currently handle dependencies that are pathed when developing, but versioned when published to PyPI.

See also https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#choosing-a-test-layout-import-rules which discusses the different source layouts for when you want to distribute your tests with the project distribution, and when you don't.

An interesting example of packaging test-helpers with a library is aiohttp, which includes its pytest fixtures in the main install, but doesn't advertise them as pytest plugins. It has a small aiohttp-pytest package (or pytest-aiohttp, I forget...) which trivially imports the relevant helpers from the main aiohttp package, and then exposes the specified entrypoints for a Pytest plugin. So to the user, it looks like aiohttp-pytest is what you install to get the pytest fixtures, but it's really only what you install to enable them.

sinoroc commented 2 years ago

You can keep the tests in the source code repository and tell Poetry to not add them to source distribution and the wheel. Isn't it good enough for that use case here?

kevinsqi commented 2 years ago

Apparently Opendoor uses editable installs / local paths in their python monorepo (as described here) and have a script that replaces development pyproject.toml with a production copy that uses version numbers: https://medium.com/@d5h/hey-steven-1215e49ad340. Might be a useful workaround.

idantene commented 2 years ago

This bothered me for internal development, and while it was extremely frustrating since neither poetry nor cleo offer any documentation on their SDK, I finally managed to create a plugin for this (pypi, git).

The idea is that with this plugin, you can have non-mutual exclusive definitions in groups, and when using the --without or --only flags, the relevant groups are dropped from dependency parsing. The plugin needs to be installed prior to any such group definition.

Installation via poetry self add poet-plugin.

I have not battle-tested this, but it works nicely for our use case with e.g.

poetry install --only prod
poetry install --only dev
poetry install --without dev
poetry install --without prod

Feedback and updates etc are welcome, hope this momentarily resolves this issue for some of us out there.

joshms123 commented 2 years ago

Hello, Just following up on this. I'd definitely like to have some way of overriding the current "pyproject.toml". Maybe something how docker handles multiple compose files? I've linked below for reference. https://docs.docker.com/compose/extends/

neersighted commented 2 years ago

Related to (but not the same as #6419).

DanCardin commented 1 year ago

I can see where this kind of request might get bogged down if you're creating potentially conflicting constraints with disjoint groups.

However for my usecase, which brought me to this issue, i feel like just deferring the install-time (poetry install) "resolution" of dependencies' until they're actually selected to be installed; would solve a/my/the problem.

And by "resolution", i mean that a path dependency will ensure the existence of the path before it checks whether or not it will ultimately be installed. Whereas for non-path dependencies, there really isn't any resolution of that sort until it's being installed (afaik).

If the lockfile is not outdated, the path must have existed at the time of the poetry lock. So the fact that it might not exist (e.g. docker builds) at install-time should be irrelevant, if it's going to be excluded for some reason (i.e. it being in a group being excluded by some specific install command (poetry install --only main)

EDIT: poet-plugin initially seemed like a solution, but it appears to not be invoked early enough to avoid the path existence check

neersighted commented 1 year ago

It seems like you have have come across the wrong issue -- you might be looking for #668 instead?

DanCardin commented 1 year ago

aha! dunno how i missed that. Although they do feel very very related. I'll go subscribe there instead :P

neersighted commented 1 year ago

To be clear to any onlookers:

joaoe commented 1 year ago

Having IDE support for import and auto completion for files within the tests/ folder should "just" work.

That's a nice to have, but the main use case in my suggestion is running stuff from the terminal or CI pipeline, and stuff just works if they import dev-only dependencies, e.g., test files importing from other test files.

TBBle commented 1 year ago

I think there's two things being conflated there. A dev-only dependency, i.e., another distribution, will "just work" when it's installed, because it's installed like any other distribution, and added to sys.paths as described by its metadata.

If you want things in your tests/ folder to be importable, making them into a distribution with a setup.py as you have been doing, is feasible.

If you do not want to make that folder a separate distribution, then you would need to manually add it to your sys.paths, e.g., with PYTHONPATH, or a pytest, poe, or tox config option if using one of those runners.

I don't think this is something Poetry should handle "magically", because:

This also seems like a very different concern from what this ticket is about, as summarised by https://github.com/python-poetry/poetry/issues/1168#issuecomment-1371454409.

rockandska commented 1 year ago

Hi,

Was looking for a similar issue before opening a "feature request" and seems this one is related to what I was looking for.

Seems that poetry, even if in dedicated groups, build the dependencies tree based on all groups. Seems fair since, depending on your needs, you install just "lint" group, or "formatter" group or two groups, with main, without main, run those groups with only one python version in contrary to the package itself etc... But, sometimes, it constrains the users and hit dependency hell even if they use a group only for test/lint etc... or doesn't let them use a recent version of a package because one of the packages require an old version for a cross dependency. As an example, flake8 and its dependency on importlib-metadata < 5

So, is there any chance to add something similar as depends_on = [] or exclusive = True for groups to let the lock file be built based on group dependency and let users have more flexibility on how the dependency is achieved ?

If not related with this issue and think it should be a separate "Feature Request", let me know.

Regards,

TBBle commented 1 year ago

Seems like a related request, since even though the original request was more focused on having different versions of the same library in different groups, e.g. one pathed at dev-time, and one semver'd at non-dev time, if the pathed version of the used library itself depends on a different set libraries than the published version, the locked package set is bifurcated based on dev/non-dev, which seems to be the crux of your use-case as well.

One concern with a bifurcated package set based on groups (which applies to dev/non-dev too) is that there's no way to represent that in a wheel generated by build, you just end up with a package where installing with both groups enabled will cause failure due to impossible-to-solve conflicts, which isn't super-kind to the user. This might be solvable by being able to exclude specific groups from the wheel metadata, as in your example, the group which pulls in flake8 could be marked to be ignored by build; that doesn't solve the problem with needing to support bifurcated locked dependency trees, but it does limit it to poetry install cases, where Poetry is able to be involved and tell the user "These two groups can't be installed together" from its own metadata.

hwmq commented 1 year ago

I have an issue that at first I thought was relevant to this ticket, but now I think not so much. It may just be a niche use case if this ticket were resolved. It is probably like a feature request for poetry-core and/or pip. I'll delete this comment later.

I have a core library and a cli library. cli depends on core and declares core in tool.poetry.dependencies as path = "/path/to/core", develop = true. I would like developers to be able to make an editable install of cli using pip (for reasons I'd rather not have to clarify; again, maybe niche).

When I pip install -e /path/to/core, I get an editable core in my user site-packages. Great.

Obtaining file:///path/to/core
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done

But when I pip install -e /path/to/cli, the editable core is removed and a non-editable core is installed instead.

Obtaining file:///path/to/cli
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Processing /path/to/core (from cli==0.0.0)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done

The workaround is pretty simple: editable install cli first, then editable install core.

I hoped that when I ran pip install -e /path/to/cli, my core dependency would either be detected as satisfied by the existing editable core install, or just be built and installed as editable again because it is marked as develop = true.

It would be nice if I could have core declared as both a non-local dependency for production, and a local path dependency for development, like how this issue describes. If I request a non-editable install with pip I might hope the non-local dependency is selected, and if I request an editable install with pip I might hope the local path dependency is selected (and built+installed as editable as well). I think this might depend on the get_requires_for_build_editable hook being able to return a local vs. non-local requirement?

To disclaim again, I'm aware this may be too opinionated/unintuitive/niche, or that I am expecting too much coordination between pip and the build backend.

TheKevJames commented 10 months ago

Some more thoughts on this in an issue which I just closed as "mostly a dupe", for those collecting more info, somewhat specific to the non-dev dependency having a specified source: #8885

luigimannoni commented 9 months ago

:warning: after a couple of months dealing with the issue the below workaround does not cope well with version pinning and generated lock files, sometimes will lead to weird behaviours in poetry. I'll keep it for record, but avoid. :warning:

I've spent days with this one, as I had a setup of 5 different libraries with intricate dependencies one to another all based on a vcs link for production deployment, that meant the libraries needed to release before I could test and play with code on the other ones and if I needed to adjust the dependency meant another release.

Now I had the "pleasure" to centralise everything into a single repo with multiple git submodules and Dockerize the entire workflow to install automatically everything, take care virtual envs and make the libraries editable without polluting too much the host system. I tried different tomls but it's frowned upon on #4460 (fair enough reasoning), and it's impossible to don't mess it up when updating dependencies versions. Then it struck me, just use an optional group for production dependency, for example:

My root ./pyproject.toml, of where I can import and play with different code

[tool.poetry.dependencies]
python = "3.10.*"
library-1 = {path = "library-1", develop = true}
library-2 = {path = "library-2", develop = true}
library-3 = {path = "library-3", develop = true}

Library 2 ./pyproject.toml:

[tool.poetry.dependencies]
python = "3.10.*"
dep1 = "^1.0.0"
dep2 = "^1.0.0"
...

[tool.poetry.group.dev.dependencies]
library-1 = { path = "../library-1", develop = true }

[tool.poetry.group.production]
optional = true

[tool.poetry.group.production.dependencies]
library-1 = { git = "ssh://git@github.com/mygit/library-1.git", rev = "v1.0.1" }

Library 3 ./pyproject.toml:

[tool.poetry.dependencies]
python = "3.10.*"
dep3 = "^1.0.0"
dep4 = "^1.0.0"
...

[tool.poetry.group.dev.dependencies]
library-1 = { path = "../library-1", develop = true }
library-4 = { path = "../library-4", develop = true }

[tool.poetry.group.production]
optional = true

[tool.poetry.group.production.dependencies]
library-1 = { git = "ssh://git@github.com/mygit/library-1.git", rev = "v1.0.1" }
library-4 = { git = "ssh://git@github.com/mygit/library-4.git", rev = "v1.1.0" }

etc...

Then on CI for each library I can run poetry install --without dev --with production --no-interaction --no-ansi --no-cache and it gets the remote revs, on local a simple poetry install --with dev uses the local editable libraries.

Of course I still need to take care to update revisions and version numbers across the tomls as waterfall before releasing, but that can be done with a script and in any case part of a normal workflow anyways.

Yes the big caveat is that people need to be able to modify the sub-projects tomls and still can't do overrides on sub-dependencies dependencies like composer does in php but that can get control back on multirepo libraries under the same ownership.

jmdeschenes commented 8 months ago

I've spent days with this one, as I had a setup of 5 different libraries with intricate dependencies one to another all based on a vcs link for production deployment, that meant the libraries needed to release before I could test and play with code on the other ones and if I needed to adjust the dependency meant another release.

Now I had the "pleasure" to centralise everything into a single repo with multiple git submodules and Dockerize the entire workflow to install automatically everything, take care virtual envs and make the libraries editable without polluting too much the host system. I tried different tomls but it's frowned upon on #4460 (fair enough reasoning), and it's impossible to don't mess it up when updating dependencies versions. Then it struck me, just use an optional group for production dependency, for example: ....

This is pretty nice!

Unfortunately, if you use dependencies from pypi, it won't work:

[tool.poetry.group.production.dependencies]
library-1 = { version = "0.1.0", source = "my-source }

Maybe poetry should not check for this?

jonapich commented 8 months ago

I wrote some tooling to help with the use case... Sorry it's using the dev-dependencies thingie:

It works pretty well. The key for the root dev project is this pydev=true.

The tooling is called stew. The relevant commands for the use case:

I'm not sure it works with the new poetry groups at all, by the way. It's also opiniated around git.

KirillShoniya commented 2 months ago

Same issue for me =-(

I develop project with few services.

Some of this services depends on our library (shared-lib).

In develop process we need to do some changes in that library in editable mode. For better development experience.

So I tried solve the problem in this way, but it's not working:

[tool.poetry.dependencies]
python = "3.11.*"

[tool.poetry.group.shared.dependencies]
shared-lib = {version = "0.0.1", source = "private-registry"}

[tool.poetry.group.dev.dependencies]
shared-lib = {path = "/path/to/shared-lib/", develop = true}

I got an error:

Incompatible constraints in requirements of my-project (0.0.1):
shared-lib @ file:///path/to/shared-lib (0.0.1)
shared-lib (==0.0.1) ; source=private-registry

This could be used like this:

# local
poetry install --without shared --with dev

# deployment
poetry install --without dev

Is any solutions for this case?

aman-gupta-doc commented 1 month ago

Hi

Any new update or development for this?