dev packages aren't installed from sub-projects

[gemmell]

Firstly, thank you for finally putting in a good solution to managing monorepos. It seems very useful thus far.

The only problem I've hit is that if I have dev dependencies in my nested packages, installing a top level package that includes that doesn't include that group.

[tool.poetry]
name = "my-package"
version = "0.0.0"
description = "A meta project for my software"
# This is a meta project, it doesn't have a package associated with it
package-mode = false
authors = ["me"]

[tool.poetry.dependencies]
python = ">=3.10, <3.12"
qb-notebook-server = {path = "apps/notebook_server", develop = true}
qb-thing-server = {path = "apps/thing_server", develop = true}

[tool.poetry-monorepo-dependency-plugin]
enable = true

[tool.poetry-dynamic-versioning]
enable = true
vcs = "git"
style = "pep440"

[build-system]
requires = ["poetry>=1.2.0b1", "poetry-dynamic-versioning"]
build-backend = "poetry_dynamic_versioning.backend"

Where those sub packages have a pyproject.toml like:

[tool.poetry]
name = "qb-thing-server"
version = "0.0.0"
description = "A RESTful server that provides ..."
authors = ["Me"]

[tool.poetry.dependencies]
python = ">=3.10, <3.12"
fastapi = "^0.109.0"
uvicorn = {extras = ["standard"], version = "^0.18.2"}
websockets = "^12.0"

[tool.poetry.group.dev.dependencies]
pytest = "^7.4.4"
pytest-xdist = "^3.5.0"
pytest-asyncio = "^0.20.3"
asgi-lifespan = ">=2.1.0"
pytest_async = "^0.1.1"
requests = "^2.31.0"
httpx = "^0.23.0"
pydantic = "^2.5.3"
anyio = "<4"

[tool.poetry-monorepo-dependency-plugin]
enable = true

[tool.poetry-dynamic-versioning]
enable = true
vcs = "git"
style = "pep440"

[build-system]
requires = ["poetry>=1.2.0b1", "poetry-dynamic-versioning"]
build-backend = "poetry_dynamic_versioning.backend"

The problem is that the dev dependencies of the nested group don't get installed. What I would like is for this plugin to be smart enough to pass down the groups to sub-packages?

For now I've worked around it by moving the dev packages to the top level pyproject.toml.

[d-ryan-ashcraft]

Thanks for trying out the monorepo plugin and for the issue!

We think this is working as designed in Poetry, though we have also found it a bit confusing. Here's how we understand Poetry's approach:

• tool.poetry.group.dev.dependencies are only used in the build of the direct package in which they are defined
• To use those dependencies during runtime from another package, you can define them as extras

This seems consistent with having each package have a definition that is independent in its own right even in the context of a monorepo structure.

Please let us know if you see it differently!

[gemmell]

So I guess the next question is how others are solving this problem. Perhaps something like this?

[tool.poetry]
name = "my-package"
version = "0.0.0"
description = "A meta project for my software"
# This is a meta project, it doesn't have a package associated with it
package-mode = false
authors = ["me"]

[tool.poetry.dependencies]
python = ">=3.10, <3.12"
qb-notebook-server = {path = "apps/notebook_server"}
qb-thing-server = {path = "apps/thing_server"}

[tool.poetry.group.dev.dependencies]
qb-notebook-server = {path = "apps/notebook_server", develop = true, extras=['dev']}
qb-thing-server = {path = "apps/thing_server", develop = true, extras=['dev']}

[tool.poetry-monorepo-dependency-plugin]
enable = true
...

The dependency in the dev group is develop = true with extras=['dev'] so in my mind this will mean that if the top package is installed with poetry install it'll install the dev groups for sub packages. The only downside being that you need to specify it twice.

[gemmell]

Doesn't seem to work :(

[d-ryan-ashcraft]

I think I misunderstood your original post. It seems like you want is to access to the same group of dev dependencies in two modules. extras would help w/ optional dependency installations - but not getting a core set of dev dependencies in multiple modules Assuming this is accurate:

• We'd recommend thinking about each module as a standalone component from a dev dependency perspective
• If you want to use the same dev dependency group in two (or more) places, you'd do something like:
  • Create a new my-package-server-dev-dependencies module
  • Add that module as a dev dependency to both my-package and qb-thing-server (and where ever else you want to use it)
  • This is consistent to the approach that would be used with Maven or similar multi-module approaches and maintains strong version control for code and matching virtual environments across the modules in the mono-repo.

[micahjsmith]

dev dependencies (and any other non-main groups) are not installed by poetry when installing path dependencies. this is explicitly not supported, and is not a problem with this plugin. you would need to use extras.

[gemmell]

Same for extras though, if I say poetry install --all-extras it doesn't install all extras from sub dependencies (which makes complete sense from the poetry perspective). Is there a way I can "pass down" the extras? I'm assuming no (by design).

In which case am I left with putting all the extras for all my sub-packages in the top level package or doing a "all-extras" package similar to the dev-dependencies package? Just FYI, I want to install all extras in my monorepo for linter CI jobs so they don't reporting missing packages (and related errors) :)

[stuaxocabinetoffice]

Ah, this is a real pain - is there an upstream poetry ticket for this ?

[d-ryan-ashcraft]

As noted above, creating a common component works well for most use cases.

We recommend using a build process to coordinate multiple components in a predictable order with predictable steps. Specifically, we use Maven, but there are numerous other choices (e.g., Pants, Bazel, Gradle), depending on your preferences and the specific set of technologies that need support. Some Python projects also try to do this with GitHub Actions or pre-commit. We discuss why we prefer Maven in this blog post that also highlights our sister project, habushu, which incorporates the poetry-monorepo-dependency-plugin into Maven builds.

We are otherwise closing this issue as support is outside the context of the poetry-monorepo-dependency-plugin.

[stuaxocabinetoffice]

Thanks for the input - I was replying on the wrong ticket: my issue wasn't around dev groups - but around extras through in general. I'd like to avoid maven if possible, as I'd rather not add the JVM as a dependency.

Apologies for the noise.