Open hozn opened 5 years ago
Just wanted to chime in here to say this is really important. Currently setting up a monorepo with python services/libraries, and need to work around these current limitations. Trying out a set up similar to what @dermidgen proposed (https://github.com/dermidgen/python-monorepo/), but it is definitely a workaround, and doesn't work for all cases.
At the end of the day, poetry supports local paths for dependencies so I think there needs to be a sane outcome when running poetry build
in this case. IMHO, it should most likely package all the local dependencies up into the built .whl
file - that's the simplest solution. If something is local, and we're building a module to be installed somewhere, it makes sense to bundle any non-external dependencies with the build, because there is no other way to get them.
The approaches I am pursuing now involve makefiles, parsing the toml to get dependencies (and transitive dependencies, etc.) that are local, and making sure I have *.whl
files for all of them, so they can be installed in the right order in a Docker image. This really needs to be handled by build tooling - i.e. poetry
.
@KholdStare I have built a solution around this problem (obtaining wheels from lock files + local dependencies) in coveo-stew.
Thought I might chime in, as I also would love to see support for optionally using local path dependencies, specifically for a "monorepo" use case.
Expanding on the "overrides file" idea from @NeilGirdhar, I feel like there is some precedent for this in the form of pip's constraints.
The same way that pip constraints files use a "subset" of the contents and syntax of requirements.txt
files, a constraints file for poetry would use a "subset" of the context and syntax of pyproject.toml
files. For instance:
if your pyproject.toml
looks like:
[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^4.1" # from your private package index, or wherever else this is published
then your constraints.toml
might would look like:
[tool.poetry.dependencies]
my-lib1 = { path = "../lib1/", develop = true }
then, whenever you wanted to work on your app/library locally, you could configure your environment with something like
poetry install --constraints constraints.toml
I'm wondering why nobody mentioned the simpler approach of Cargo (rust), where path dependencies are converted to version dependencies when building/publishing (freezing the version of the path dependency at the time of the build). This would also avoid specifying/updating the required versions on every update. Maybe I've missed something.
We can also be more precise on constraints by changing the meaning of version
when using path dependencies. I've a really early idea (and probably broken), for example here if the local dependency has version 1.2.3
:
# Simple constraints:
dep = { path = "<path>", version = ">=x" } # actual constraint: >=1
dep = { path = "<path>", version = ">=x.x" } # actual constraint: >=1.2
dep = { path = "<path>", version = "^x.x" } # actual constraint: ^1.2
dep = { path = "<path>", version = "^x.x, !=1.2.0" } # we can also use classic constraints
# By default, this constraints can be used to allow only patch updates (according to sementic versionning):
dep = { path = "<path>", version = "~x.x.x" } # actual constraint: ~1.2.3
dep = { path = "<path>", version = ">=x.x.x, <x.{x+1}.0" } # equivalent to the previous one, less readable, and {x+1} syntax is ugly imo
What do you think about this idea? I can try to work on a proof-of-concept pull request if you find that useful.
@mindstorm38 In your example, are you storing the path dependencies in the shared repository file "pyproject.toml", which is visible to all users and developers? The point of local path dependencies is for an individual developer to override the settings in this file.
(Incidentally, since I'm commenting anyway, the nice thing about @poocat's idea of a "constraints.toml" is that you can have various sets of local override sets. Nice idea.)
@NeilGirdhar My idea is inspired from a personal project where the local path is only valid within the repository (it's a subproject structure, with dependencies between subprojects). It's also used like this with Cargo workspaces we can also use git submodules (even if I don't like such dependency management, I also know that a workspace issue is being discussed, but for me, it's not directly related to the current one and is not required for it).
For me, dependencies within the pyproject.toml
should not point outside the project's root, maybe overrides.toml
is more suitable for what @poocat suggested, and can be used as a user-defined overrides to the pyproject.toml
, but it seems off-topic for this issue, or I'm wrong about the issue?
it's a subproject structure, with dependencies between subprojects
Maybe I've misunderstood, but I don't think that makes sense for a project that is uploaded to PyPI.
For me, dependencies within the pyproject.toml should not point outside the project's root
You mean that dependencies like this shouldn't exist?
Maybe I've misunderstood, but I don't think that makes sense for a project that is uploaded to PyPI.
In my case, I'm working with a single git repository, with a core python package and many optional add-ons packages, that I want to be distributed as well, but separately from the core package and dependent to it. It's like poetry
and poetry-core
, but within a single repo (it would be too complicated for my little project to make multiple repos, and many devs are in this case I guess).
You mean that dependencies like this shouldn't exist?
No, of course, I meant path-dependencies.
In my case, I'm working with a single git repository,
Right, but poetry needs to support projects that aren't "monorepos", and also projects that are uploaded to PyPI.
No, of course, I meant path-dependencies.
The path dependencies that I and poobar are talking about do point outside the repository. This allows a Poetry "editable mode" akin to pip install -e
.
Ok, I see. Just to clarify, my idea would allow at the same time monorepos, multi-repos and publishing to PyPI, that's the main goal of it, because for now poetry lack of monorepos support imo.
Do you think that this idea could fit into poetry, or contributing is a lost of time? I like monorepos, but this might be the wrong issue to discuss it, and in my case it's quite specific to path dependencies...
I'm not sure if this is the right issue. If you look at the very first comment, the dependencies listed are:
my-lib1 = { path = "../lib1/", develop = true }
So it looks like this person wants to depend on things outside the repository in a kind of development ("editable") mode.
As for monorepos, I don't know much about them, but maybe start a new issue explaining what you want to poetry to do, and link this issue as related?
This is a good idea, thanks I'll work on that!
@mindstorm38 See #2270 maybe.
I know this one, but I disagree with this approach of a single pyproject.toml for a whole repo. But that's not the point because they don't talk about path dependencies and how to resolve them at build time.
https://github.com/python-poetry/poetry/discussions/3646 is the discussion going about this.
Hi.
My use case is simpler than the OPs.
In my pyproject.toml
I have
[tool.poetry]
name = "lettuce"
version = "0.0.0"
description = "Salad is good for you"
[tool.poetry.dev-dependencies]
t-lettuce = {path = "tests", develop = true}
So during a regular non-dev build, the lettuce
shall always be installed in my environment.
During a dev install, I want my unit test module to also be installed as a python module, such that individual test files can have a from t_lettuce import helper_function
.
So far, this pyprojet.toml has worked for me, plus a setup.py
file in the tests folder.
However, I'd like there not to be a requirement to put a setup.py
or pyproject.toml
inside the tests folder, since these are just stub files to fool poetry install.
@joaoe Out of curiosity, why are your tests an entire package? Why not use pytest?
@joaoe I think you might be asking for something that's not feasible in Python packaging. By my understanding, the set of "what packages are in the project dist" cannot be affected by install-time options like 'extras', as the included package list is defined at sdist/wheel creation time, not installation time.
A close match would be a feature request for the format
field of the packages
list to support another value, e.g. editable
(adding to current sdist
and wheel
), which then would include that package only during an editable install (which is not the same as a 'dev' install, but might work for your use-case). The idea being that something like this would work:
[tool.poetry]
name = "lettuce"
version = "0.0.0"
description = "Salad is good for you"
packages = [
{ include = "lettuce", from = "src" },
{ include = "t-lettuce", from = "tests", format = "editable" },
]
and then when installed in an editable install, e.g. as the root package in a Poetry venv, or when installing as a pathed develop dependency from some other package; t_lettuce
would exist as a module, but would not be included in either an sdist
or a wheel
built from the same source.
That would be a Poetry-specific behaviour, and might depend on how the Poetry implementation of PEP-660 functions: Poetry-core supports PEP-660 already, but Poetry itself still manages editable installs directly, not using poetry-core's PEP-660 support, so such a feature might need to be implemented in two places.
Out of curiosity, why are your tests an entire package? Why not use pytest?
I'm using pytest. Want we want is simply for the tests/
folder to be populated in sys.path
(without having to do that explicitly in code) so any test file can import stuff from other test files, and in a way that works for IDEs and tools to provide auto complete and other kinds of checks.
Plus, I see other scenarios where I could have an extra folder with scripts and utilities which shall not be distributed, but still in the path when the project's virtualenv is active.
A close match would be a feature request for the
format
field of thepackages
list to support another value, e.g.editable
(adding to currentsdist
andwheel
), which then would include that package only during an editable install (which is not the same as a 'dev' install, but might work for your use-case).
That seems like a decent suggestion. But now that I think about it, tagging a package folder as a dev dependency would somehow allow for the feature of having say "production" and "development" builds of a package, where the "development" could include stuff like unit tests. Or is that already possible ?
@joaoe Having IDE support for import and auto completion for files within the tests/
folder should "just" work. The only thing you need to do is to make your tests forlder a proper python package by including __init__.py
files in all folders. With that PyCharm and VSCode (just tested) allow for full auto completion and imports from one file in the test folder to another using either relative or absolute import path (import config from .base_test_file
or import config from tests.base_test_file
. No further configuration required.
So I think your usecase does not need any support from Poetry
But now that I think about it, tagging a package folder as a dev dependency would someone allow for the feature of having say "production" and "development" builds of a package, where the "development" could include stuff like unit tests.
This is the thing that the Python packaging ecosystem doesn't handle in any way. There's no "variations" of a project in this sense, they would be two separate projects. The closest is the 'extras' mechanism which only changes dependencies at install-time, so you could have your test helpers (and tests, I guess...) in a separate package that your package depends on with a 'dev' extra.
However, then you're back to the core issue of this ticket, which is that Poetry can't currently handle dependencies that are pathed when developing, but versioned when published to PyPI.
See also https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#choosing-a-test-layout-import-rules which discusses the different source layouts for when you want to distribute your tests with the project distribution, and when you don't.
An interesting example of packaging test-helpers with a library is aiohttp
, which includes its pytest fixtures in the main install, but doesn't advertise them as pytest plugins. It has a small aiohttp-pytest
package (or pytest-aiohttp
, I forget...) which trivially imports the relevant helpers from the main aiohttp package, and then exposes the specified entrypoints for a Pytest plugin. So to the user, it looks like aiohttp-pytest
is what you install to get the pytest fixtures, but it's really only what you install to enable them.
You can keep the tests in the source code repository and tell Poetry to not add them to source distribution and the wheel. Isn't it good enough for that use case here?
Apparently Opendoor uses editable installs / local paths in their python monorepo (as described here) and have a script that replaces development pyproject.toml with a production copy that uses version numbers: https://medium.com/@d5h/hey-steven-1215e49ad340. Might be a useful workaround.
This bothered me for internal development, and while it was extremely frustrating since neither poetry
nor cleo
offer any documentation on their SDK, I finally managed to create a plugin for this (pypi, git).
The idea is that with this plugin, you can have non-mutual exclusive definitions in groups, and when using the --without
or --only
flags, the relevant groups are dropped from dependency parsing. The plugin needs to be installed prior to any such group definition.
Installation via poetry self add poet-plugin
.
I have not battle-tested this, but it works nicely for our use case with e.g.
poetry install --only prod
poetry install --only dev
poetry install --without dev
poetry install --without prod
Feedback and updates etc are welcome, hope this momentarily resolves this issue for some of us out there.
Hello, Just following up on this. I'd definitely like to have some way of overriding the current "pyproject.toml". Maybe something how docker handles multiple compose files? I've linked below for reference. https://docs.docker.com/compose/extends/
Related to (but not the same as #6419).
I can see where this kind of request might get bogged down if you're creating potentially conflicting constraints with disjoint groups.
However for my usecase, which brought me to this issue, i feel like just deferring the install-time (poetry install
) "resolution" of dependencies' until they're actually selected to be installed; would solve a/my/the problem.
And by "resolution", i mean that a path dependency will ensure the existence of the path before it checks whether or not it will ultimately be installed. Whereas for non-path dependencies, there really isn't any resolution of that sort until it's being installed (afaik).
If the lockfile is not outdated, the path must have existed at the time of the poetry lock
. So the fact that it might not exist (e.g. docker builds) at install-time should be irrelevant, if it's going to be excluded for some reason (i.e. it being in a group being excluded by some specific install command (poetry install --only main
)
EDIT: poet-plugin initially seemed like a solution, but it appears to not be invoked early enough to avoid the path existence check
It seems like you have have come across the wrong issue -- you might be looking for #668 instead?
aha! dunno how i missed that. Although they do feel very very related. I'll go subscribe there instead :P
To be clear to any onlookers:
Having IDE support for import and auto completion for files within the
tests/
folder should "just" work.
That's a nice to have, but the main use case in my suggestion is running stuff from the terminal or CI pipeline, and stuff just works if they import dev-only dependencies, e.g., test files importing from other test files.
I think there's two things being conflated there. A dev-only dependency, i.e., another distribution, will "just work" when it's installed, because it's installed like any other distribution, and added to sys.paths
as described by its metadata.
If you want things in your tests/
folder to be importable, making them into a distribution with a setup.py as you have been doing, is feasible.
If you do not want to make that folder a separate distribution, then you would need to manually add it to your sys.paths
, e.g., with PYTHONPATH
, or a pytest, poe, or tox config option if using one of those runners.
I don't think this is something Poetry should handle "magically", because:
test/
path added to sys.paths;This also seems like a very different concern from what this ticket is about, as summarised by https://github.com/python-poetry/poetry/issues/1168#issuecomment-1371454409.
Hi,
Was looking for a similar issue before opening a "feature request" and seems this one is related to what I was looking for.
Seems that poetry, even if in dedicated groups, build the dependencies tree based on all groups.
Seems fair since, depending on your needs, you install just "lint" group, or "formatter" group or two groups, with main, without main, run those groups with only one python version in contrary to the package itself etc...
But, sometimes, it constrains the users and hit dependency hell even if they use a group only for test/lint etc... or doesn't let them use a recent version of a package because one of the packages require an old version for a cross dependency.
As an example, flake8 and its dependency on importlib-metadata
< 5
So, is there any chance to add something similar as depends_on = []
or exclusive = True
for groups to let the lock file be built based on group dependency and let users have more flexibility on how the dependency is achieved ?
If not related with this issue and think it should be a separate "Feature Request", let me know.
Regards,
Seems like a related request, since even though the original request was more focused on having different versions of the same library in different groups, e.g. one pathed at dev-time, and one semver'd at non-dev time, if the pathed version of the used library itself depends on a different set libraries than the published version, the locked package set is bifurcated based on dev/non-dev, which seems to be the crux of your use-case as well.
One concern with a bifurcated package set based on groups (which applies to dev/non-dev too) is that there's no way to represent that in a wheel generated by build
, you just end up with a package where installing with both groups enabled will cause failure due to impossible-to-solve conflicts, which isn't super-kind to the user. This might be solvable by being able to exclude specific groups from the wheel metadata, as in your example, the group which pulls in flake8
could be marked to be ignored by build
; that doesn't solve the problem with needing to support bifurcated locked dependency trees, but it does limit it to poetry install
cases, where Poetry is able to be involved and tell the user "These two groups can't be installed together" from its own metadata.
I have an issue that at first I thought was relevant to this ticket, but now I think not so much. It may just be a niche use case if this ticket were resolved. It is probably like a feature request for poetry-core and/or pip. I'll delete this comment later.
I have a core library and a cli library. cli depends on core and declares core in tool.poetry.dependencies
as path = "/path/to/core", develop = true
. I would like developers to be able to make an editable install of cli using pip (for reasons I'd rather not have to clarify; again, maybe niche).
When I pip install -e /path/to/core
, I get an editable core in my user site-packages. Great.
Obtaining file:///path/to/core
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Installing backend dependencies ... done
Preparing editable metadata (pyproject.toml) ... done
But when I pip install -e /path/to/cli
, the editable core is removed and a non-editable core is installed instead.
Obtaining file:///path/to/cli
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... done
Processing /path/to/core (from cli==0.0.0)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
The workaround is pretty simple: editable install cli first, then editable install core.
I hoped that when I ran pip install -e /path/to/cli
, my core dependency would either be detected as satisfied by the existing editable core install, or just be built and installed as editable again because it is marked as develop = true
.
It would be nice if I could have core declared as both a non-local dependency for production, and a local path dependency for development, like how this issue describes. If I request a non-editable install with pip I might hope the non-local dependency is selected, and if I request an editable install with pip I might hope the local path dependency is selected (and built+installed as editable as well). I think this might depend on the get_requires_for_build_editable
hook being able to return a local vs. non-local requirement?
To disclaim again, I'm aware this may be too opinionated/unintuitive/niche, or that I am expecting too much coordination between pip and the build backend.
Some more thoughts on this in an issue which I just closed as "mostly a dupe", for those collecting more info, somewhat specific to the non-dev dependency having a specified source
: #8885
I've spent days with this one, as I had a setup of 5 different libraries with intricate dependencies one to another all based on a vcs link for production deployment, that meant the libraries needed to release before I could test and play with code on the other ones and if I needed to adjust the dependency meant another release.
Now I had the "pleasure" to centralise everything into a single repo with multiple git submodules and Dockerize the entire workflow to install automatically everything, take care virtual envs and make the libraries editable without polluting too much the host system. I tried different tomls but it's frowned upon on #4460 (fair enough reasoning), and it's impossible to don't mess it up when updating dependencies versions. Then it struck me, just use an optional group for production dependency, for example:
My root ./pyproject.toml
, of where I can import and play with different code
[tool.poetry.dependencies]
python = "3.10.*"
library-1 = {path = "library-1", develop = true}
library-2 = {path = "library-2", develop = true}
library-3 = {path = "library-3", develop = true}
Library 2 ./pyproject.toml
:
[tool.poetry.dependencies]
python = "3.10.*"
dep1 = "^1.0.0"
dep2 = "^1.0.0"
...
[tool.poetry.group.dev.dependencies]
library-1 = { path = "../library-1", develop = true }
[tool.poetry.group.production]
optional = true
[tool.poetry.group.production.dependencies]
library-1 = { git = "ssh://git@github.com/mygit/library-1.git", rev = "v1.0.1" }
Library 3 ./pyproject.toml
:
[tool.poetry.dependencies]
python = "3.10.*"
dep3 = "^1.0.0"
dep4 = "^1.0.0"
...
[tool.poetry.group.dev.dependencies]
library-1 = { path = "../library-1", develop = true }
library-4 = { path = "../library-4", develop = true }
[tool.poetry.group.production]
optional = true
[tool.poetry.group.production.dependencies]
library-1 = { git = "ssh://git@github.com/mygit/library-1.git", rev = "v1.0.1" }
library-4 = { git = "ssh://git@github.com/mygit/library-4.git", rev = "v1.1.0" }
etc...
Then on CI for each library I can run poetry install --without dev --with production --no-interaction --no-ansi --no-cache
and it gets the remote revs, on local a simple poetry install --with dev
uses the local editable libraries.
Of course I still need to take care to update revisions and version numbers across the tomls as waterfall before releasing, but that can be done with a script and in any case part of a normal workflow anyways.
Yes the big caveat is that people need to be able to modify the sub-projects tomls and still can't do overrides on sub-dependencies dependencies like composer does in php but that can get control back on multirepo libraries under the same ownership.
I've spent days with this one, as I had a setup of 5 different libraries with intricate dependencies one to another all based on a vcs link for production deployment, that meant the libraries needed to release before I could test and play with code on the other ones and if I needed to adjust the dependency meant another release.
Now I had the "pleasure" to centralise everything into a single repo with multiple git submodules and Dockerize the entire workflow to install automatically everything, take care virtual envs and make the libraries editable without polluting too much the host system. I tried different tomls but it's frowned upon on #4460 (fair enough reasoning), and it's impossible to don't mess it up when updating dependencies versions. Then it struck me, just use an optional group for production dependency, for example: ....
This is pretty nice!
Unfortunately, if you use dependencies from pypi, it won't work:
[tool.poetry.group.production.dependencies]
library-1 = { version = "0.1.0", source = "my-source }
Maybe poetry should not check for this?
I wrote some tooling to help with the use case... Sorry it's using the dev-dependencies thingie:
It works pretty well. The key for the root dev project is this pydev=true.
The tooling is called stew. The relevant commands for the use case:
stew bump
will bump all lock files in the monorepostew pull-dev-dependencies
will analyze the dev dependencies of all the projects and import them into the root's pyprojectstew check-outdated
will report on outdated lock files, or outdated dev dependencies.stew build
is similar to pip download
, it bundles everything so you can install offline. Docs here.I'm not sure it works with the new poetry groups at all, by the way. It's also opiniated around git.
Same issue for me =-(
I develop project with few services.
Some of this services depends on our library (shared-lib).
In develop process we need to do some changes in that library in editable mode. For better development experience.
So I tried solve the problem in this way, but it's not working:
[tool.poetry.dependencies]
python = "3.11.*"
[tool.poetry.group.shared.dependencies]
shared-lib = {version = "0.0.1", source = "private-registry"}
[tool.poetry.group.dev.dependencies]
shared-lib = {path = "/path/to/shared-lib/", develop = true}
I got an error:
Incompatible constraints in requirements of my-project (0.0.1):
shared-lib @ file:///path/to/shared-lib (0.0.1)
shared-lib (==0.0.1) ; source=private-registry
This could be used like this:
# local
poetry install --without shared --with dev
# deployment
poetry install --without dev
Is any solutions for this case?
Hi
Any new update or development for this?
Question
I believe this issue is related to #668 -- or some of the comments therein -- but I believe my workflow / use case is also a little different from the origin of that issue.
Here's the background / what I'm trying to accomplish:
Here's how I would do this with setuptools / pip:
my-app
e.g.) has arequirements.txt
file that has loose enough versions of my shared libraries so that it'll be satisfied by the next version that I'm working on. For example, ifmy-app
will use an updatedmy-lib1
version 4.1 andmy-lib2
version 1.12.3, I might have the following inmy-app/requirements.txt
:pip install -r requirements.txt && python setup.py develop
(Sometimespip install -e .
would also work, but the behavior was a little more consistent actually usingpython setup.py develop
. Argh, the mess that is python packaging !?!)pip install -r requirements.txt
in my main application; because I'd already donepython setup.py develop
I'd have installed (symlinked) in the right versions to satisfy formy-lib1
andmy-lib2
.my-app
was pushed to build server, it'd find the versions it needed.This all works reasonably well, though the initial environment setup is a pain. I'm really hoping to move to Poetry but I also feel that I "need" an equivalent to the above.
I can use "path dependencies" to point to my local shared libraries, but then these aren't going to work when it actually goes to build on the CI server (or on anyone else's workstation if they organize their files differently):
If I specify the same dependency in
[tool.poetry.dependencies]
and[tool.poetry.dev-dependencies]
, it seems to always pick the one from the non-dev deps:But even if this did work, ideally the solution wouldn't involve me using path dependencies at all, since (in this case) these are specific to how my workstation is setup -- a coworker may organize things differently.
So, is there a current solution to solving this problem? Perhaps I've missed something in reading the docs. Thanks for any help!