Support for mutually exclusive dependencies across groups (e.g. path dependencies during development only)

[hozn]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Question

I believe this issue is related to #668 -- or some of the comments therein -- but I believe my workflow / use case is also a little different from the origin of that issue.

Here's the background / what I'm trying to accomplish:

• I have a large modular python project. It is generally a main application with dependencies on other components that we maintain -- shared libraries, essentially.
• Adding a new feature in the main application often involves changing one or more of these shared libraries.
• During development, I'd like to be able to experiment with different code in those shared libraries and integration test it with my main application before actually writing tests for / committing / tagging / pushing / etc. the changes to those shared libraries.

Here's how I would do this with setuptools / pip:

• Ensure that my main application (my-app e.g.) has a requirements.txt file that has loose enough versions of my shared libraries so that it'll be satisfied by the next version that I'm working on. For example, if my-app will use an updated my-lib1 version 4.1 and my-lib2 version 1.12.3, I might have the following inmy-app/requirements.txt:

  my-lib1 ^4.1
  my-lib2 ^1.12.3

• Setup a virtualenv for my top-level application and activate it.
• Change into each shared library directory and run pip install -r requirements.txt && python setup.py develop (Sometimes pip install -e . would also work, but the behavior was a little more consistent actually using python setup.py develop. Argh, the mess that is python packaging !?!)
• Finally I'd run pip install -r requirements.txt in my main application; because I'd already done python setup.py develop I'd have installed (symlinked) in the right versions to satisfy for my-lib1 and my-lib2.
• After finishing development, I'd package up the shared libraries and deploy them to our registry; then when my-app was pushed to build server, it'd find the versions it needed.

This all works reasonably well, though the initial environment setup is a pain. I'm really hoping to move to Poetry but I also feel that I "need" an equivalent to the above.

I can use "path dependencies" to point to my local shared libraries, but then these aren't going to work when it actually goes to build on the CI server (or on anyone else's workstation if they organize their files differently):

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }
# THIS ONLY WORKS ON MY WORKSTATION

If I specify the same dependency in [tool.poetry.dependencies] and [tool.poetry.dev-dependencies], it seems to always pick the one from the non-dev deps:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^4.1"
my-lib2 = "^1.12.3"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }

# IT ALWAYS TRIES TO FIND MATCHING VERSIONS ON THE PYPI SERVER, DOESN'T USE THE PATHS

But even if this did work, ideally the solution wouldn't involve me using path dependencies at all, since (in this case) these are specific to how my workstation is setup -- a coworker may organize things differently.

So, is there a current solution to solving this problem? Perhaps I've missed something in reading the docs. Thanks for any help!

[ankostis]

The solution to this existing problem requires some kind tof "redirection": the "path dependencies" contain the exact path, while another "type of dependency" would allow each user to configure ("redirect to") its exact path.

Actually, not even a new type of dependency is needed, as long as the user can override and re-direct any regular dependency to its own path.

[hozn]

@ankostis , I agree that allowing the user to override -- e.g. with commandline arguments -- during the install step would meet my requirements here. Something like:

poetry install --path-dep ../path/to/my-lib1 --path-dep ../path/to/my-lib2

It would probably still be required that the dependency at that local path matched the version specified in my pyproject.toml file.

[avanzal]

I'm currently facing the same challenges as described.

Basically, I have a number of library packages and a number of app packages using those libraries, and would like to:

1. Use poetry to install dependencies during local development (where app and library packages are on disk)
2. Use poetry to build dists for the library packages

The problem seems to be a disconnect between path dependencies, poetry install, and poetry build:

1. While developing, it's pretty handy to use path dependencies (instant updates without build+publish, edits across multiple libraries at once, etc); but
2. If I specify any path dependencies in pyproject, I can't do the required poetry build (doesn't allow path dependencies)
3. If I specify only non-path dependencies in pyproject, poetry install / poetry update will fetch from the configured repository instead of local dirs, which is counter to point 1

I've tried (and failed) on some workarounds:

• I can't use poetry add --path, because the dependency already specified in pyproject. Also it's pretty painful to do this for each library.
• I tried adding the libraries as non-path dependencies to tools.poetry.dependencies and as path dependencies to tools.poetry.dev-dependencies, hoping poetry build and poetry install would pick from these separately, but this doesn't work (poetry build still fails)

Found some partial / potential workarounds, though none of these cover all requirements:

• Use poetry run pip install -e ... for local dependencies, and write some scripting automate this somewhat as it would be pretty painful to do by hand for all the connected libraries.
• Use only non-path dependencies in the library pyprojects, and add all local libraries (including sub-dependencies) as path dependencies to the app's pyproject. Running poetry install on the app uses local disk via the specified path dependencies. However, in my use case each library package has its own test suite, so they also need to do a poetry install, and since they're now non-path dependencies this doesn't work with local development.
• Each developer could run their own local PyPi server and publish their dev packages to it. Really not ideal though, especially if you're changing the libraries a lot.

And also:

• It would be really useful if path dependencies also supported version = "..." (eg. xyz-log = {path="../xyz-log", version="^1.0.0"}), would help with branched code etc.

I think this problem would be solved if there was a configurable package resolver - eg. you could configure it to first look up the package name in a particular directory and if it exists then install it as path dependency, else go look for a git repo <base-url>/<package-name>.git, else go look in PyPi, etc - whatever the user configures. This is on similar lines to the command line parameter idea of poetry install --path-dep ... from the comments

[bersace]

I'm facing this issue while maintaining two poetry-managed libraries. I did the following:

~/src/dep$ poetry install
~/src/dep$ cd ../main/
~/src/main$ poetry install
~/src/main$ echo $HOME/src/dep >> $VIRTUAL_ENV/lib/python3.6/site-packages/easy-install.pth
~/src/main$ python -c 'import dep; print(dep.__file__)'
/home/.../src/deps/dep.py
~/src/main$

That's not the best solution, but still fire and forget.

[pbryan]

I too am in a similar situation. I'm substituting a dependency (e.g. x = "^1.3") with a path dependency (x = { path = "../x") in pyproject.toml, which works. Ideally, there would be a method of overriding any dependency (or dev-dependency) in a "poetry-installed" project without modifying pyproject.toml (maybe in some kind of "override" file).

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[hozn]

Um, yes this is still relevant! Unless something has changed this is still just as broken/incomplete as it was when the issue was created.

[hozn]

@sdispater , could we get feedback on this issue?

[freddrake]

I'm using path dependencies similar to some of the others who've followed up is this issue already, but am interesting in having those dependencies built and installed into the virtual env just like anything else, not as "editable" installs using .egg-link metadata. My goal is to be able to use the virtual env as input to a process that builds a platform package (.deb in my case).

Being able to control the type of installation separately from the type of source would be quite handy.

[niander]

Any update on this?

[idmitrievsky]

Can anyone weigh in on what are the obstacles for this functionality, so we could begin working on eliminating them?

[abn]

@hozn Your use case sounds very similar to one I have porposed at #2270. It would be good to get more eyes on that one and possibly feedback as a way to solve this.

Additionally, there is also #755 which might be another solution once implemented. However, that will only work if you are okay working with VCS copies. This would also imply that you can simply use something like git-submodules, which might not be ideal.

@idmitrievsky at the moment, I do not think there is a specific solution proposed here. If #2270 would be a solution that would be awesome, it is something I do want to work on soon once we have some feedback.

If poetry were to allow for dev dependencies to override specified dependencies, this would sort of break the locking mechanism itself. A possible compromise might be to allow for a local configuration that allows for the pre-pending of certain paths in the project's virtualenv without using it to determine dependencies or update lockfile etc. If that is something that is desirable, I do think that will sit well inside a plugin once the plugin system is completed.

[idmitrievsky]

@abn thank you for your response! I will keep an eye on the plugin system then.

[applied-mathematician]

Add me to the list of those facing this issue. My team is working on a large django project that incorporates multiple python apps (eventually installed on multiple nodes) and shared local libraries. While I already sold them on moving to poetry, getting a dev environment set up currently seems to more of a hassle than it's worth. I'd prefer to wait than use the work-arounds others have mentioned for simulating an editable install.

@abn I like your #2270 proposal and would like to help with any feedback/planning to get things rolling ASAP. My gitlab ticket of moving our project to poetry is going to hang until one of the proposed solutions gets implemented.

[marcelmindemann]

Would this work? I'd create a seperate pyproject.toml in a new folder which lists all of my main package's in-company dependencies as editable path dependencies. I then run poetry install followed by poetry shell. In this shell, with the venv activated, I then do my development in my main package. The dependencies in my main package's pyproject.toml can stay regular version dependencies, which allows poetry build. Since my shell already has a venv activated, poetry run commands will import the editable dependencies.

[misterjoa]

Hello everyone, we are also facing this as we have a monorepo and want to be able to build dev packages locally and use defined versions of packages in prod. Seems related to https://github.com/python-poetry/poetry/issues/2046

[janhurst]

I'm also facing a similar challenge working with a "main" application that uses a shared library that I have added as a Git submodule. Apologies in advanced I'm pretty new to poetry, coming from conda and manual venv.

I want to be able to work on the shared library, and interact with it's Git repo, but a CI/CD produced version of the shared library from my private Azure Artifacts feed when I build my application (in a Docker container).

I had expected behaviour similar to what I think the OP suggested where the local path was used when I poetry install when working in my dev environment, but a repo/feed version was used when I poetry install --no-dev in my Docker build.

Using something like:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^0.0.1"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "./my-lib1-in-a-git-submodule/", develop = true }

I think this partially mitigates the issue the OP mentions about co-workers organizing their projects differently.... I have the luxury of "forcing" a project layout by using a Git submodule, but regardless I think this would still be useful?

[kesiraa]

How about having separate toml files for various environments. ? i.e pyproject.dev.toml, pyproject.prod.toml This will also allow us to test newer python versions (i.e 3.9 in dev to test, while 3.8 is in the prod)

[micahscopes]

If you haven't tried it, yarn has a nice feature for this called yarn link. It could be nice to have something like this in poetry!

[briggySmalls]

@janhurst I have exactly the same issue that you do. Our production docker containers use poetry to build a wheel before installing it in a fresh image. Currently the shared library in the dev-dependencies causes this build step to fail.

[mrijken]

I have made https://pypi.org/project/poetry-dev/ in order to support this use case.

[francoposa]

I would really like this to support absolute paths, so we can reliably set it up the same way inside and outside of a dockerfile:

(learn-spark) [~/repos/learn-spark]% poetry add /usr/local/spark/python 

[ValueError]
'/usr/local/spark/python' does not start with '/Users/franco/repos/learn-spark'

So for now I'm using relative path, which won't be the same as the docker container when it gets built...

(learn-spark) [~/repos/learn-spark]% poetry add ../../../../usr/local/spark/python 

[abn]

@francojposa you could try using the url dependency. file:///usr/local/spark/python. However, I am guessing you are aware of the potential breakages this could introduce as this is not really conisdered portable.

[francoposa]

@abn That doesn't work. This isn't a total dealbreaker, but a pain. Do you think supporting absolute paths would be a relatively simple addition/would the project be open to a PR?

It seems that even though it gets declared with a relative path, Poetry ultimately understands it as an absolute path. When I switched from the local path requirement to just a standard pyspark requirement and ran poetry lock it printed out

- Updating pyspark (3.0.0 /usr/local/spark/python -> 3.0.1)

Use case is to run the same Poetry requirements during both local dev and in a Docker container that already has Spark baked in.

For vanilla pip/requirements.txt, we just pip install -e /usr/local/spark/python/.

[danmur]

It's not ideal but I used an optional dependency for my build process:

[tool.poetry.dependencies]
my-package = { version = "^0.1.0", optional = true }

[tool.poetry.dev-dependencies]
my-package = {path = "../my-package"}

[tools.poetry.extras]
full = ["my-package"]

poetry install in dev, and pip install myotherpackage[full] to deploy

[francoposa]

I don't want to derail this thread, I think my issue is more accurately described by https://github.com/python-poetry/poetry/issues/1692

[freddrake]

I don't want to derail this thread, I think my issue is more accurately described by #1692

This isn't really about using absolute paths, but whether path dependencies can be used in particular circumstances.

[kkawabat]

I'm also facing a similar challenge working with a "main" application that uses a shared library that I have added as a Git submodule. Apologies in advanced I'm pretty new to poetry, coming from conda and manual venv.

I want to be able to work on the shared library, and interact with it's Git repo, but a CI/CD produced version of the shared library from my private Azure Artifacts feed when I build my application (in a Docker container).

I had expected behaviour similar to what I think the OP suggested where the local path was used when I poetry install when working in my dev environment, but a repo/feed version was used when I poetry install --no-dev in my Docker build.

Using something like:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^0.0.1"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "./my-lib1-in-a-git-submodule/", develop = true }

I think this partially mitigates the issue the OP mentions about co-workers organizing their projects differently.... I have the luxury of "forcing" a project layout by using a Git submodule, but regardless I think this would still be useful?

I am trying to do something similar to this

[tool.poetry.dependencies]
mylibdep = {git = "git@gitlab.com:mygit/mylibdep.git", tag = "1.49.0"}

[tool.poetry.dev-dependencies]
mylibdep = {path = "../mylibdep", develop = true}

but I am getting this error when trying to poetry lock

Because mylib depends on both mylibdep (1.49.0 git tag 1.49.0) and mylibdep (1.49.0 ../mylibdep), version solving failed.

I think the problem is that poetry install installs both mylibdep and causes a conflict. Is there a way to prevent the production dependency from installing conditioned on whether -no-dev is on or not? Looking at the doc it seems like I can use environment-markers but don't know poetry enough.

Something like below to prevent conflict?:

[tool.poetry.dependencies]
mylibdep = {git = "git@gitlab.com:mygit/mylibdep.git", tag = "1.49.0",  markers = "POETRY_FLAG_NO_DEP"}

[tool.poetry.dev-dependencies]
mylibdep = {path = "../mylibdep", develop = true}

[dorranh]

@kkawabat, IIRC the issue is that poetry dev and standard dependencies form a kind of super set, so the lock command fails since you have two conflicting versions of the same package defined (one as the git dependency and as a local dependency). Are you just installing the package in the docker image build or are you trying to build a wheel? If only the former, have you tried using only the path dependency and ensuring that your CI script checkouts both the repo and the submodule? I think that should get you what you're after - my colleague had to do something similar to work with git submodules.

[sinoroc]

The following technique seems to work.

[tool.poetry.dependencies]
python = "^3.8"
foosball-lib = "*"

[tool.poetry.dev-dependencies]
pytest = "^5.2"
foosball-lib = {path = "../foosball-lib", develop = true}

See this repo for an example: https://github.com/sinoroc/foosball

What issues do you encounter with this technique?

[piotr-kopacki]

@sinoroc This will always install the foosball-lib from the path directory, even with the --no-dev flag.

[sinoroc]

This will always install the foosball-lib from the path directory, even with the --no-dev flag.

I just tested it and confirmed it. That is true, and that is strange. This is not necessarily what I would have expected. I also tried after having removed the poetry.lock lockfile beforehand, and the result was the same.

$ poetry --version
Poetry version 1.1.4
$ poetry install --no-dev
Updating dependencies
Resolving dependencies... (1.5s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing foosball-lib (0.1.0 /home/sinoroc/workspace/foosball/foosball-lib)

[TBBle]

This last problem, when dev/non-dev overlap for the same dependency, is probably the same issue as #2837.

[Kaiser1989]

Having the exact same problem. One just do not want to publish a module while developing. This is a total breaker for using poetry and developing frameworks/APIs for python. The complete python language is missing this feature.

There must be a difference between developing and distribution. A relative path in distribution doesn't even make sense. We cannot expect the user to have some module at some specific path.

Vice verca, it doesn't even make sense to use a versioned module while developing. The workflow is:

• Do changes
• Do tests
• Commit & Publish

and not

• Do changes
• Commit & Publish
• Do tests

Tbh, i don't know any other language that doesn't provide this basic feature ...

[sinoroc]

Aside

To confirm what @Kaiser1989 just wrote:

A relative path in distribution doesn't even make sense.

As far as I know, PyPI would reject any distribution containing a path dependency (and some other things as well, basically anything of the form Library @ something if I remember correctly).

[sinoroc]

I think some people use the path dependencies in "production", because they have a mono-repo or something like that. I can not claim to have totally understood that use case, but I think some people do rely on path dependencies outside of the development steps.

[TBBle]

Our current project indeed plans (or at least, it's the leading plan right now) to use the relative path dependencies in production, but, in that flow, we never package anything up, but use poetry install --no-dev to populate a container image, partly because relative path dependencies make no sense in an sdist or wheel.

This is the reason I was looking at #2837, so that --no-dev resulted in a self-contained site-packages directory, rather than symliking the source from outside, and we could just duplicate it from the builder image to the runtime image, where the latter does not need Poetry or its dependencies installed.

It would also be an good option, if this ticket is resolved first, to be able have our production image-build pipeline turn each of our libraries into sdists, and then have our non-leaf projects pull them like normal (non-path, non-git) dependencies. In our case, they'd be pulling from a local PyPI index directory rather than a PyPI index we had published to, but other than that, it looks like I understand the original request's functionality, except that being a monorepo, the path is never wrong, I just don't want it to apply if it's only in the dev-dependencies.

tl;dr: A monorepo workflow doesn't make "packages with path dependencies" a good idea; but it can make use of "production with path dependencies", e.g., #2837 wants "production with non-editable path dependencies".

[sinoroc]

@TBBle I don't think installing and using poetry in containers meant for production is a good idea either. You probably should generate all your wheels beforehand and then in the image for the production container you should install the wheels with something like pip (not poetry). At least that would be my take on it.

---

edit Re-reading your comment, maybe that is not what you meant exactly. Advice still stands anyway.

[TBBle]

Indeed, I'd like to not have Poetry in the final containers, and generally I can avoid that with a multi-stage build, using Poetry to create a virtualenv, and then copying the virtualenv into a new container image based on the same starting Python.

The workflow of generating wheels beforehand and installing them using pip for production is what I want, but I need to be able to do that while also having Poetry in dev mode using editable installs of the required libraries in our monorepo, and editable installs of their required libraries from our monorepo, etc.

That's basically what I thought this ticket was requesting.

Given this for some service:

[tool.poetry.dependencies]
python = "~=3.7.3"
# Due to #2837, this is treated as `develop = true`, that's a different issue.
my-common-lib = { path = "../libs/common/", develop = false }

[tool.poetry.dev-dependencies]
my-common-lib = { path = "../libs/common/", develop = true }
my-pytest-fixture-collection = { path = "../libs/pytest-fixtures/", develop = true }

And poetry build would produce a setup.py with

install_requires=['my-common-lib'],
# I haven't checked, I am assuming this is what `tool.poetry.dev-dependencies` translates into in setup.py
extras_require={'dev': ['my-pytest-fixture-collection']}

with the onus on me to ensure I also make poetry build output of my-common-lib and my-pytest-fixture-collection available to pip when it installs this.

Looking at that example, in short, I think I get what I want if poetry build simply stripped path and develop fields from dependencies before translating them into setup.py.

---

Reading the original issue again, I realise the author was talking about poetry install --no-dev, not poetry build, so perhaps I'm just on the wrong ticket. >_<

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^4.1"
my-lib2 = "^1.12.3"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }

seems like a similar problem to #2837, in that we want to lock based on all the dependencies (--no-dev should not affect the lock file), but poetry install --no-dev should not be affected by path or develop settings (or anything else except the version spec used for locking) in dev-dependencies.

[jonapich]

2693 also uses the trick of duplicating the libs in dev/nondev to obtain a mix between development convenience and publishing to public servers.

In my case, all the projects are local to the repository so the paths are identical for everyone.

[mguryev]

I have recently ran into this problem, and solved the problem by using pip to install dependencies in docker environment. I would like to leave this idea here in case someone else is trying to build docker image from the monorepo

My setup:

• python code is stored in monorepo
• path based dependencies are used for local development

pyproject.toml

[tool.poetry.dependencies]
my-lib1 = "^0.0.1"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "../lib1/", develop = true }

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

When building docker image, I add all of the monorepo code, then parse pyproject.toml to find local paths and build a wheel for local dependencies.

Then I am pointing pip at the wheels for local dependencies when installing dependencies in the docker image.

#!/usr/bin/env sh

set -eu

export PROJECT_DIR="$(pwd)"
export WHEELHOUSE_DIR="$1"

echo '=== building wheelhouse for project $PROJECT_DIR ==='
test -d "$WHEELHOUSE_DIR"

echo "=== building local dependencies ==="
grep -o 'path[^,}]*' pyproject.toml | while read -r line ; do
    echo "Processing $line"
    DEPENDENCY="$(echo $line | sed -E 's/.*"([^"]*)".*/\1/')"
    (cd $DEPENDENCY && pip wheel --no-deps --wheel-dir "$WHEELHOUSE_DIR" .)
done

echo "=== initializing virtual environment ==="
pip wheel download --wheel-dir "$WHEELHOUSE_DIR" poetry-core>=1.0.0
pip install --find-links "$WHEELHOUSE_DIR" .

[finswimmer]

I wrote about it here.

[awilkins]

Yeah, now running into this :

If you do a poetry build in a project with a path dependency, the setup.py in your sdist looks like this :

install_requires = \
['aws_xray_sdk>=2.6.0,<3.0.0',
 'boto3>=1.16.60,<2.0.0',
 'my-library @ ../my_library',
 'jsonpickle>=1.5.0,<2.0.0',
 'pycognito>=0.1.5,<0.2.0',
 'requests>=2.25.1,<3.0.0']

That just isn't going to work deployed to a PyPi repo.

This is a set of libraries that are in a mono-repo. It makes sense to have them with develop=true path dependencies during test, but not when built into a distribution.

Possibilities...?

For builds

1. Make it an error to do poetry build for a project with local path dependencies (and explain why)
2. Read the version from the path target, and set the required version to ^that.version in the built setup.py / METADATA
3. Permit an explicit version requirement which gets used for build output, alongside a path which gets used for local install
   • Make it an error if the path-based dependency isn't compatible with the version specifier
   • Presently mixing a version specifier and a path is considered an error
4. Markers : add a marker that's only present at build-time?

Tried :

Defining the same lib as a path in dev-dependencies doesn't work ; seems like the dev-dependencies entry squashes the entry in the lock ; if you move the directory out of the way, poetry install --no-dev fails. This also then makes poetry update fail.

[defreng]

I want to bring in a different idea here (heavily inspired by Golang modules)

I think a simple command line switch for poetry install could solve this problem. Something like:

poetry install \
   --replace 'mypackage:{path = "../foosball-lib", develop = true}' \
   --replace 'anotherpackage: ...'

The switch would basically just replace all entries of mypackage in the poetry lockfile with the version given in the argument.

What do you think? Would this cover all usecases?

[sinoroc]

@defreng I really like the idea of being able to bypass the dependency resolution (and/or the index, the cache, etc.) for some specific dependencies. I have suggested this somewhere already, don't remember which discussion it was (I didn't know it was possible in Golang). I don't know if it would cover all use cases.

[dermidgen]

I've created an example repo that shows how to do this; without additional poetry support.
https://github.com/dermidgen/python-monorepo/

[NeilGirdhar]

@defreng I like your idea, but maybe instead of a command-line switch, a local configuration file would be easier. Something like pyproject_overrides.toml, and then keep the same format as pyproject.toml. That way you can set up your local configuration, and maybe override other things in the future. For example, the number of jobs for pytest would be another reasonable local machine override.

So, to set up a local override, you could use the new poetry install --editable flag along with a --local flag to put the configuration in pyproject_overrides.toml

[DmytroSokhach]

Thanks all for the answers. Quite a short thread. 😶 I seem to have different behaviour locally and using Dockerfile: Running poetry install -vv --no-dev writes lock file and then stucks forever on:

Writing lock file

Finding the necessary packages for the current system

I'm following this comment with latest poetry: https://github.com/python-poetry/poetry/issues/1168#issuecomment-659234244 which makes sense to me. Thanks @janhurst pyproject.toml for parent module contains:

[tool.poetry.dependencies]
configurationmanager = "~0.1.0"

[tool.poetry.dev-dependencies]
configurationmanager = { path = "../libs/python-configurationmanager", develop = true }

[mkarbo]

There seems to be a quite general issue for organisations/individuals working with monorepos and using Poetry.

I think there is a good opportunity to improve poetry, by e.g., allowing for flagging to build from local directories (as per comment above, with the wheelhouse, but part of Poetry CLI somehow) or building some external tooling for handling this logic, instead of custom docker/shell scripts which seem to be appear on every result when googling "poetry local path dependency monorepo build"

[chickeaterbanana]

My use cases would be met with something like:

if current_command == "install" and  "no_dev" not in command_option:
    precedence = option.get("precedence", "local")
elif "path" in option and "version" in option:
    precedence = option.get("precedence", "remote")
elif "path" in option and "version" not in option:
    precedence = "local"
elif "path" not in option and "version" in option:
    precedence = "remote"

if current_command == "build" and precedence == "remote":
    """
        lookup in repositories/sources if the package isn't found - use the logic of `precedence == local`
        if the package is found ignore `path`
    """
elif current_command == "build" and precedence == "local":
    """
        lookup if the wheel is already build in path and copy the wheel to this folder if found, otherwise
        build sdist/wheel in path and place this wheel and sdist also in folder dist

        specifiy the version and package-name as dependency and _not_ the path

        if the specified dependency is also poetry than this logic should run recursively. The result should be all wheels/sdist of all
        relative dependencies are in the folder `dist`.
    """
elif current_command == "install" and precedence == "local":
    """
        current behavior - install as develop if develop is specified...
    """
elif current_command == "install" and precedence == "remote":
    """
        lookup in repositories/sources if the package isn't found - use the logic of `precedence == local`
        if the package is found ignore `path`
    """

Last but not least the export command should return only the package name and version as dependencies. Otherwise pip would try to install the path.

This would allow me to run poetry build and end up with all wheels I need to install the package except the only-remote dependencies. At the development I can have path and develop to have fast development cycles.

This would allow me to have something like this as dockerfile:

FROM bitnami/python:3.9.9-debian-10-r14 as builder
RUN groupadd --gid 1001 non-root-group && \
    useradd --gid 1001 --uid 1001 non-root-user

WORKDIR /app

ENV PIP_DEFAULT_TIMEOUT=100 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PIP_NO_CACHE_DIR=1 \
    POETRY_VERSION=1.1.12

RUN install_packages \
        libpq-dev \
        build-essential \
    && \
    pip install "poetry==$POETRY_VERSION"

COPY pyproject.toml poetry.lock ./
RUN poetry export -f requirements.txt > requirements.txt
COPY . .
RUN poetry build -f wheel
RUN mkdir -p /venv && chown -R 1001:1001 /venv

USER 1001

RUN python -m venv /venv
RUN /venv/bin/pip install wheel

RUN /venv/bin/pip install -r ./requirements.txt && /venv/bin/pip install dist/*.whl

FROM bitnami/python:3.9.9-prod-debian-10-r14 as final

RUN groupadd --gid 1001 non-root-group && \
    useradd --gid 1001 --uid 1001 non-root-user && \
    mkdir -p /venv && \
    chown -R 1001:1001 /venv

USER 1001

COPY --from=builder /venv /venv
COPY container/docker-entrypoint.sh ./

ENTRYPOINT ["./docker-entrypoint.sh"]