poetry export - should it include optional extras by default?

dazza-codes commented 3 years ago

What is the intention for poetry export with respect to default behavior for optional dependencies? It seems to include them in the output requirements.txt file.

poetry export --without-hashes --format requirements.txt --output requirements.txt

For example, if scipy is an optional extra, it should not be in requirements.txt (right?)

scipy = {version = "^1.5.3", optional = true}

But, there it is:

scipy==1.5.4; python_version >= "3.6"

Given there are additional command options to include optional extras, it suggests the default behavior is to exclude all optional dependencies (unless explicitly requested with an --extras {x}).

Could it be that a dev-dep for e.g. seaborn has a dep on scipy and the poetry export is not filtering out a dep of a dev-dep when it exports? As in python-poetry/poetry#3448 ?

sinoroc commented 3 years ago

Can you use poetry show --tree to figure out what depends on scipy?

dazza-codes commented 3 years ago

To replicate it, try

[tool.poetry]
name = "tmp"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.7"
numpy = {version = "^1.19.4", optional = true}

[tool.poetry.extras]
# updated to address comment 1 below
numpy = [ "numpy"]

[tool.poetry.dev-dependencies]
pandas = "^1.1.5"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

$ poetry export --without-hashes --format requirements.txt --output requirements.txt
$ cat requirements.txt 
numpy==1.19.4; python_version >= "3.6"

Is this desired behavior or should it only include numpy when the export explicitly requests it with an --extras numpy option?

$ poetry show -t
numpy 1.19.4 NumPy is the fundamental package for array computing with Python.
pandas 1.1.5 Powerful data structures for data analysis, time series, and statistics
├── numpy >=1.15.4
├── python-dateutil >=2.7.3
│   └── six >=1.5 
└── pytz >=2017.2

$ poetry show -t --no-dev
numpy 1.19.4 NumPy is the fundamental package for array computing with Python.
pandas 1.1.5 Powerful data structures for data analysis, time series, and statistics
├── numpy >=1.15.4
├── python-dateutil >=2.7.3
│   └── six >=1.5 
└── pytz >=2017.2

It should exclude pandas from the latter show, right?

sinoroc commented 3 years ago

Looks to me like your pyproject.toml is incomplete. In poetry marking dependencies as optional (i.e. as part of an extra) is a two steps thing. And in the example you are showing here I do not see the [tool.poetry.extras] section of the pyproject.toml. So I do not know if your numpy is really being considered as part of an optional extra by poetry or not.

Maybe read the doc again:

https://python-poetry.org/docs/pyproject/#extras

thedrow commented 3 years ago

Mine is definately complete, however the constraint is still exported despite not specified as an extra during poetry export.

See https://github.com/celery/jumpstarter/blob/5b690eb91890d46e3a0cacbb26d616ef3f6ec215/pyproject.toml

sinoroc commented 3 years ago

@thedrow But transitions is a mandatory dependency of transitions-anyio, which is a mandatory dependency of your own project, so of course both are always exported, right?

thedrow commented 3 years ago

The bug here is that the requirement to clone it from git is included even though the extra was not specified. Some extras may require higher versions of a dependency to support something. In this case I used it to install the newest bug fixes I needed but in the near future when transitions fixes the bugs it has when using newer graphziv versions, I'd require a higher version of transitions if state machine diagrams support is required.

dazza-codes commented 3 years ago

Sorry if the example did not have the extras clause, but it was quickly drafted an an example. In other projects where this actually arose and motivated the issue, the extras clause is defined. Just modify the example to test the functionality please. (I will update the example in the original note above.)

finswimmer commented 3 years ago

@dazza-codes: numpy is a dependency by pandas. For some reason the dev-dependencies are export, which shouldn't be the case without providing the --dev flag. :thinking:

dazza-codes commented 3 years ago

Good luck with this. I've since abandoned poetry export and replaced it with workflows that use poetry build and pip installs from the package.

dimbleby commented 2 years ago

this looks like an error in locking rather than in exporting:

[[package]]
name = "numpy"
version = "1.21.1"
description = "NumPy is the fundamental package for array computing with Python."
category = "main"
optional = false
python-versions = ">=3.7"

I guess it's optional in "main", not optional in "dev", and this has been mis-combined to make it not optional in "main".

Not sure what the lockfile should look like here, is the format rich enough to express the desired behaviour?

Andarius commented 2 years ago

@dimbleby helped me fix this issue, running poetry update lock then poetry export -f requirements.txt did not export my extra packages

kbakk commented 1 year ago

Posted this (incorrectly it seems) in another issue, but it looks to belong here. It seems to repeat what others are mentioning, but it's another data point so it could be useful.

I believe an issue I have with dev dependency moto and optional main dependency boto3 leads to the same problem as described here.

Given the state of this gist: https://gist.github.com/kbakk/bd0ab6720d24bcefa65fda09eaa7e93a/c4d4b1ef8917d3d1ab6c637d7b4764e16a6fb61e

pyproject.toml

[tool.poetry]
name = "ppe-42"
version = "0.1.0"
description = ""
authors = ["Kristoffer Bakkejord <kbakk@users.noreply.github.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
boto3 = {version = "^1.26.7", optional = true}

[tool.poetry.extras]
b = ["boto3"]

[tool.poetry.group.dev.dependencies]
moto = "^4.0.9"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Current state of boto3:

$ grep -B1 -A5 'name = "boto3"' poetry.lock
[[package]]
name = "boto3"
version = "1.26.7"
description = "The AWS SDK for Python"
category = "main"
optional = false
python-versions = ">= 3.7"

Then run poetry remove moto ... boto3 will become optional:

grep -B1 -A5 'name = "boto3"' poetry.lock
[[package]]
name = "boto3"
version = "1.26.7"
description = "The AWS SDK for Python"
category = "main"
optional = true
python-versions = ">= 3.7"

It seems to be this code that's responsible for including/excluding boto3:

https://github.com/python-poetry/poetry-plugin-export/blob/679e6a71d6cc7f3bf3f9d801c5b66bb8f13bd68b/src/poetry_plugin_export/walker.py#L95-L99

However, the evaluation becomes (when both boto3 and moto are in the lock file):

>>> extra_package_names is not None
True
>>> package.optional
False
>>> package.name not in extra_package_names
True
>>> extra_package_names is not None and ( package.optional and package.name not in extra_package_names)
False

So it misses the continue statement.

aucampia commented 1 year ago

From #174

`poetry export --only=main` includes optional dependencies that are not optional in a dependency group

Poetry version: 1.3.1
Python version: 3.11
OS version and name: Fedora Linux 37
pyproject.toml: https://gist.githubusercontent.com/aucampia/885b6f7cf51f4e1986627f66ee06254f/raw/faae8d71a6cc24bcf254821a3474cb3cca192f33/pyproject.toml

[x] I am on the latest stable Poetry version, installed using a recommended method.
[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] I have consulted the FAQ and blog for any relevant entries or release notes.
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

When I run poetry export --only=main I don't expect to see anything in the output except things in the main dependency group that are not optional, however, if a dependency is optional in the main dependency group, and not optional in another dependency group, like dev, then it will be included in poetry export --only=main.

So in essence, given my pyproject.toml, I would expect

$ pipx run --spec=poetry==1.3.1 --python=python3.11 poetry export --without-hashes --only=main
click==8.1.3 ; python_version >= "3.11" and python_version < "4.0"
colorama==0.4.6 ; python_version >= "3.11" and python_version < "4.0" and platform_system == "Windows"
typer==0.7.0 ; python_version >= "3.11" and python_version < "4.0"

However, what I actually get is:

$ pipx run --spec=poetry==1.3.1 --python=python3.11 poetry export --without-hashes --only=main
click==8.1.3 ; python_version >= "3.11" and python_version < "4.0"
colorama==0.4.6 ; python_version >= "3.11" and python_version < "4.0" and platform_system == "Windows"
pydantic==1.10.4 ; python_version >= "3.11" and python_version < "4.0"
typer==0.7.0 ; python_version >= "3.11" and python_version < "4.0"
typing-extensions==4.4.0 ; python_version >= "3.11" and python_version < "4.0"

If I remove pydantic = {version = "^1.10.4"} from [tool.poetry.group.dev.dependencies] then I do get the output I expect, but then pydantic is no longer installed when you run poetry install.

Complete repo that can be used to reproduce can be found here.

python-poetry / poetry-plugin-export