group-specific updates to extras work: unintended, or undocumented?

[a-recknagel]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

When looking up the docs on how to define extras, the final box says in very clear words, that you can only define the packages that you can list in a [tool.poetry.extras] entry within the [tool.poetry.dependencies]/[tool.poetry.dependencies.main] section:

The dependencies specified for each extra must already be defined as project dependencies. Dependencies listed in dependency groups cannot be specified as extras.

This is not 100% true, I can re-define dependencies as optional in a group, and if I install that group plus the extra, poetry will consider the dependency as it is defined in the group.

For example, I have an extra snowflake which installs snowflake-connector-python with its pandas extra. During development, I'd like to install its secure-local-storage extra as well. I can express this with

[tool.poetry]
name = "group-extras"
version = "0.1.0"
description = ""
authors = ["Arne <me@arne.com>"]
readme = "README.md"
packages = [{include = "group_extras"}]

[tool.poetry.dependencies]
python = "^3.10"
snowflake-connector-python = {version = ">=3.2.1", optional = true, extras = ["pandas"]}

[tool.poetry.group.dev.dependencies]
snowflake-connector-python = {version = "*", optional = true, extras = ["secure-local-storage"]}

[tool.poetry.extras]
snowflake = ["snowflake-connector-python"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

And it will work just fine.

I expected a syntax error due to the option=true in a non-main group, but it resolves

$ poetry lock
Updating dependencies
Resolving dependencies... [time passes]
Writing lock file

This installs snowflake-connector-python[pandas]

$ poetry install --only main --extras snowflake --sync
Installing dependencies from lock file

Package operations: 27 installs, 0 updates, 0 removals

  • Installing pycparser (2.21)
  [...]
  • Installing pandas (2.0.3)
  [...]
  • Installing snowflake-connector-python (3.3.1)

Installing the current project: group-extras (0.1.0)

This installs snowflake-connector-python[pandas,secure-local-storage], note that pandas is not getting removed

$ poetry install --with dev --extras snowflake --sync
Installing dependencies from lock file

Package operations: 7 installs, 0 updates, 0 removals

  • Installing jeepney (0.8.0)
  • Installing more-itertools (10.1.0)
  • Installing zipp (3.17.0)
  • Installing importlib-metadata (6.8.0)
  • Installing jaraco-classes (3.3.0)
  • Installing secretstorage (3.3.3)
  • Installing keyring (24.2.0)

Installing the current project: group-extras (0.1.0)

This installs snowflake-connector-python[pandas] again, removing all dependencies from the secure-local-storage. At this point, I'd say that this works in a sane manner.

$ poetry install --only main --extras snowflake --sync
Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 7 removals

  • Removing importlib-metadata (6.8.0)
  • Removing jaraco-classes (3.3.0)
  • Removing jeepney (0.8.0)
  • Removing keyring (24.2.0)
  • Removing more-itertools (10.1.0)
  • Removing secretstorage (3.3.3)
  • Removing zipp (3.17.0)

Installing the current project: group-extras (0.1.0)

If it were a bug or unintended behavior, I'd have expected that one dependency specification overwrites the other one. But since the listed extras from the dependency's definition in different groups are combined correctly, it feels like something that is safe to use. It's quite useful too, so shouldn't it be documented?

[MajorDallas]

I also just discovered this behavior in a similar search. In my case, I have a dependency on Celery and a dev dependency on Pytest. Celery has an extra for a pytest plugin that I would like to use but not include in a build for a production deployment.

I did more or less the same thing:

[tool.poetry.dependencies]
celery = {version="^5.3.6", extras=["redis", "tblib", "auth"]}

[tool.poetry.group.dev.dependencies]
celery = {version="^5.3.6", extras=["pytest"]}

My poetry.lock was unmodified after locking this configuration. Since I'm not clear on how poetry chooses what to include in a build I'm not 100% sure if that's what I wanted, but it's probably fine 🙂

So, +1 for documenting this use-case. I'd also propose a way to declare if a specific extra belongs to a different group from the parent package. Maybe something like:

[tool.poetry.dependencies.celery]
version = "^5.3.6"
extras = [
    {name="pytest", group="dev"},
    "redis",
    "tblib",
    "auth",
]

In the end, declaring a package twice isn't that much of a pain. There's a potential for version conflict if one copies verbatim the version spec like I did (rather than using "*" like OP) and then later changes one but not the other (in my example, going to "^6" while the other is "^5.3.6"). Even that would be an easy fix, but I could see it being one of those things that's easy to forget about and miss while reading the file, then spend a couple hours trying to figure out where this mysterious version conflict is coming from when it clearly says "^6" in the file.