When using self-referential extras in pyproject.toml, the package is added to the requirements

[rfsan]

If I run

pip-compile --annotation-style=line --no-header --all-extras --output-file=pyproject.lock pyproject.toml

with the following pyproject.toml

[project]
name = "my-pkg"
version = "0.0.1"

[project.optional-dependencies]
tests = ["pytest"]
dev = ["ruff", "my-pkg[tests]"]

The result includes a self-reference

iniconfig==2.0.0          # via pytest
my-pkg[tests] @ file:///home/...  # via my-pkg (pyproject.toml)
packaging==23.2           # via pytest
pluggy==1.3.0             # via pytest
pytest==7.4.2             # via my-pkg, my-pkg (pyproject.toml)
ruff==0.0.292             # via my-pkg (pyproject.toml)

I would expect

iniconfig==2.0.0          # via pytest
packaging==23.2           # via pytest
pluggy==1.3.0             # via pytest
pytest==7.4.2             # via my-pkg,(pyproject.toml)
ruff==0.0.292             # via my-pkg (pyproject.toml)

[WhyNotHugo]

"my-pkg[tests]" means "this package with the tests extras. my-pkg is then added to extras because you're explicitly requesting it.

[WhyNotHugo]

I understand what you're actually trying to do, but I'm not sure that it's possible to express.

Personally, I would just include pytest as a dev dependency.

[webknjaz]

Alternative opinion: I dislike the use of extras for test/dev deps — they are basically your public API so why would you expose them to the non-contributing end-users? I think, it's semantically wrong. With these deps sets, you usually aim to describe virtualenvs where you run your stuff (test, dev, docs), rather than your project. So why not just use regular requirements as they were designed?

[webknjaz]

Though, I've seen this @ file:///home/ thing in other places (editable + relative?) and thought it'd be nice to fix it (there's difference between -e . and . for some reason..

[rfsan]

The example I gave above was intended to be easy to replicate.

Here is a real world example using pandas pyproject.toml

[project.optional-dependencies]
test = ['hypothesis>=6.46.1', 'pytest>=7.3.2', 'pytest-xdist>=2.2.0', 'pytest-asyncio>=0.17.0']
performance = ['bottleneck>=1.3.4', 'numba>=0.55.2', 'numexpr>=2.8.0']
computation = ['scipy>=1.8.1', 'xarray>=2022.03.0']
fss = ['fsspec>=2022.05.0']
aws = ['s3fs>=2022.05.0']
gcp = ['gcsfs>=2022.05.0', 'pandas-gbq>=0.17.5']
excel = ['odfpy>=1.4.1', 'openpyxl>=3.0.10', 'python-calamine>=0.1.6', 'pyxlsb>=1.0.9', 'xlrd>=2.0.1', 'xlsxwriter>=3.0.3']
parquet = ['pyarrow>=7.0.0']
feather = ['pyarrow>=7.0.0']
hdf5 = [# blosc only available on conda (https://github.com/Blosc/python-blosc/issues/297)
        #'blosc>=1.20.1',
        'tables>=3.7.0']
spss = ['pyreadstat>=1.1.5']
postgresql = ['SQLAlchemy>=1.4.36', 'psycopg2>=2.9.3']
mysql = ['SQLAlchemy>=1.4.36', 'pymysql>=1.0.2']
sql-other = ['SQLAlchemy>=1.4.36']
html = ['beautifulsoup4>=4.11.1', 'html5lib>=1.1', 'lxml>=4.8.0']
xml = ['lxml>=4.8.0']
plot = ['matplotlib>=3.6.1']
output-formatting = ['jinja2>=3.1.2', 'tabulate>=0.8.10']
clipboard = ['PyQt5>=5.15.6', 'qtpy>=2.2.0']
compression = ['zstandard>=0.17.0']
consortium-standard = ['dataframe-api-compat>=0.1.7']
all = ['beautifulsoup4>=4.11.1',
       # blosc only available on conda (https://github.com/Blosc/python-blosc/issues/297)
       #'blosc>=1.21.0',
       'bottleneck>=1.3.4',
       'dataframe-api-compat>=0.1.7',
       'fastparquet>=0.8.1',
       'fsspec>=2022.05.0',
       'gcsfs>=2022.05.0',
       'html5lib>=1.1',
       'hypothesis>=6.46.1',
       'jinja2>=3.1.2',
       'lxml>=4.8.0',
       'matplotlib>=3.6.1',
       'numba>=0.55.2',
       'numexpr>=2.8.0',
       'odfpy>=1.4.1',
       'openpyxl>=3.0.10',
       'pandas-gbq>=0.17.5',
       'psycopg2>=2.9.3',
       'pyarrow>=7.0.0',
       'pymysql>=1.0.2',
       'PyQt5>=5.15.6',
       'pyreadstat>=1.1.5',
       'pytest>=7.3.2',
       'pytest-xdist>=2.2.0',
       'pytest-asyncio>=0.17.0',
       'python-calamine>=0.1.6',
       'pyxlsb>=1.0.9',
       'qtpy>=2.2.0',
       'scipy>=1.8.1',
       's3fs>=2022.05.0',
       'SQLAlchemy>=1.4.36',
       'tables>=3.7.0',
       'tabulate>=0.8.10',
       'xarray>=2022.03.0',
       'xlrd>=2.0.1',
       'xlsxwriter>=3.0.3',
       'zstandard>=0.17.0']

It would be possible to create a group of optional dependencies. For example, sql = ['pandas[postgresql]', 'pandas[mysql]', 'pandas[sql-other]']. This way you don't have to write more than once a dependency constrains (as they did in the all extra) which is prone to errors because you could update one and forget to update the other.

The "my-pkg[tests]" self-reference is because pip expects this syntax (it's supported since pip 21.1 and they are working on documenting it, link). Self-references are also been discussed here

[WhyNotHugo]

It would be possible to create a group of optional dependencies. For example, sql = ['pandas[postgresql]', 'pandas[mysql]', 'pandas[sql-other]']. This way you don't have to write more than once a dependency constrains (as they did in the all extra) which is prone to errors because you could update one and forget to update the other.

AFAIK, there's no way to express dependencies this way in pyproject.toml.

The "my-pkg[tests]" self-reference is because pip expects this syntax (it's supported since pip 21.1 and they are working on documenting it, https://github.com/pypa/pip/issues/11296). Self-references are also been discussed here

pip installs my-pkg too if you use this syntax. You want "my-pkg[tests]" minus my-pkg. I am under the impression that pip does not have a way to express this.

[pe224]

pip installs my-pkg too if you use this syntax.

This is understandable and ok, since installing my-pkg[dev] is always expected to install my-pkg. Hence I wouldn't mind the somewhat redundant line

my-pkg  # via my-pkg (pyproject.toml)

but having a reference to the local filesystem as indicated in the issue description

[...]
my-pkg[tests] @ file:///home/...  # via my-pkg (pyproject.toml)
[...]

breaks compatibility of the resulting file with pip install -r/-c on any other machine.

Since this syntax presumably is the only declarative way (within pyproject.toml) for nested extras, it might need to be special-cased within pip-tools(?)

[q0w]

but having a reference to the local filesystem

Pip supports only this way.

[pe224]

As a workaround for now, I found applying the following CLI options works

pip-compile --unsafe-package my-pkg --no-allow-unsafe

Like this, the breaking reference of my-pkg to the local filesystem will not be included, which saves me from manually editing the resulting requirements.txt file after running pip-compile.

Note that this command now pins the previously "unsafe" packages (like pip or setuptools), but excluding these packages from being pinned will be deprecated anyways (see #989). Hence the --no-allow-unsafe is added to future-proof the command.

[AndydeCleyre]

Another thought in the arena of workarounds (sorry):

For me, requirements files still trump pyproject.toml as a source of truth, so I use the former and some scripting to (re)populate the latter. For the first example in this issue, I'd have dev-requirements.in and tests-requirements.in:

$ <dev-requirements.in
ruff
-r tests-requirements.in

$ <tests-requirements.in
pytest

Then running the pypc function from my pip-tools frontend, inject new values into pyproject.toml. It calls a bit of Python using tomlkit, you can steal the logic:

$ pypc
$ <pyproject.toml

[project]
name = "my-pkg"
version = "0.0.1"

[project.optional-dependencies]
tests = ["pytest"]
dev = ["pytest", "ruff"]

[webknjaz]

You want "my-pkg[tests]" minus my-pkg. I am under the impression that pip does not have a way to express this.

FYI there's a draft PEP 735 attempting to address this.