`poetry` mismatch with `pip` on optional dependencies install when not defined in extras.

[daquintero]

Description

Hi, not sure if I should raise this with pip-setuptools specifically, so thanks for the help guiding me on who to point this towards. Not sure if this is the expected behaviour either?

We recently ran into an issue with one of our projects on this. In summary, in our pyproject.toml reproducible at the state below we have:

https://github.com/flexcompute/tidy3d/blob/6f6371606e83b18eca2b73579f02ff6a4e4de357/pyproject.toml#L104

The flax dependency is defined as an optional dependency in the pyproject.toml, however, we forgot to add it onto an extra group. What happened was that anytime we would do: pip install -e . at that commit state for example, it would also install flax alongside the core base dependencies. This was with:

± |develop → origin {4} U:1 ✗| → python --version
Python 3.10.13
pip                23.0.1

Running: pip install -e .:

Installing collected packages: pytz, zipp, urllib3, tzdata, typing-extensions, toolz, toml, six, PyYAML, pyroots, pyparsing, pyjwt, pygments, protobuf, pillow, packaging, numpy, nest_asyncio, msgpack, mdurl, locket, kiwisolver, joblib, jmespath, importlib_resources, idna, future, fsspec, fonttools, etils, cycler, cloudpickle, click, charset-normalizer, certifi, annotated-types, absl-py, shapely, scipy, requests, python-dateutil, pydantic-core, partd, opt-einsum, ml-dtypes, markdown-it-py, importlib-metadata, h5py, contourpy, autograd, tensorstore, rich, responses, pydantic, pandas, matplotlib, jaxlib, h5netcdf, dask, botocore, xarray, s3transfer, jax, orbax-checkpoint, chex, boto3, optax, >>>>>>>flax<<<<<<<<, tidy3d

However, when I install the package with poetry install, flax is not installed:

Installing dependencies from lock file

Package operations: 53 installs, 0 updates, 1 removal

  - Removing setuptools (70.0.0)
  - Installing six (1.16.0)
  - Installing python-dateutil (2.9.0.post0)
  - Installing idna (3.7)
  - Installing numpy (1.26.4)
  - Installing packaging (24.1)
  - Installing pygments (2.18.0)
  - Installing pyyaml (6.0.1)
  - Installing typing-extensions (4.12.2)
  - Installing certifi (2024.6.2)
  - Installing charset-normalizer (3.3.2)
  - Installing scipy (1.13.1)
  - Installing urllib3 (2.2.2)
  - Installing fsspec (2024.6.0)
  - Installing mdurl (0.1.2)
  - Installing requests (2.32.3)
  - Installing toolz (0.12.1)
  - Installing zipp (3.19.2)
  - Installing annotated-types (0.7.0)
  - Installing jmespath (1.0.1)
  - Installing markdown-it-py (3.0.0)
  - Installing pydantic-core (2.18.4)
  - Installing botocore (1.34.128)
  - Installing click (8.1.7)
  - Installing contourpy (1.2.1)
  - Installing cycler (0.12.1)
  - Installing fonttools (4.53.0)
  - Installing future (1.0.0)
  - Installing kiwisolver (1.4.5)
  - Installing locket (1.0.0)
  - Installing pillow (10.3.0)
  - Installing pydantic (2.7.4)
  - Installing pyparsing (3.1.2)
  - Installing pytz (2024.1)
  - Installing rich (13.7.1)
  - Installing tzdata (2024.1)
  - Installing autograd (1.6.2)
  - Installing cloudpickle (3.0.0)
  - Installing h5py (3.11.0)
  - Installing importlib-metadata (7.1.0)
  - Installing matplotlib (3.9.0)
  - Installing pandas (2.2.1)
  - Installing partd (1.4.2)
  - Installing s3transfer (0.10.1)
  - Installing boto3 (1.34.128)
  - Installing dask (2024.6.0)
  - Installing h5netcdf (1.0.2)
  - Installing joblib (1.4.2)
  - Installing pyjwt (2.8.0)
  - Installing pyroots (0.5.0)
  - Installing responses (0.25.3)
  - Installing shapely (2.0.4)
  - Installing toml (0.10.2)
  - Installing xarray (2024.6.0)

Installing the current project: tidy3d (2.7.2)

Workarounds

When flax is included in an extra group, then this is not a problem as both pip and poetry work as expected.

Poetry Installation Method

pipx

Operating System

Ubuntu 22.04

Poetry Version

Poetry (version 1.8.2)

Poetry Configuration

± |develop → origin {4} U:1 ✗| → poetry config --list
cache-dir = "/home/daquintero/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
repositories.jaxsource.url = "https://storage.googleapis.com/jax-releases/jax_releases.html"
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/daquintero/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

https://github.com/flexcompute/tidy3d/blob/6f6371606e83b18eca2b73579f02ff6a4e4de357/pyproject.toml

Poetry Runtime Logs

Inconsistency with `pip` mainly.

[daquintero]

Actually closing this as when I did the first search of issues, this issue seems to discussing this even if the title of the issue could be improved so it's easier to find this https://github.com/python-poetry/poetry/issues/8322

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.