tweag / FawltyDeps

Python dependency checker
Other
201 stars 14 forks source link

[conda] Support Pixi's use of pyproject.toml to declare conda dependencies? #453

Closed jherland closed 1 month ago

jherland commented 2 months ago

Thanks to @ctcjab's comment for highlighting Pixi's use of pyproject.toml in conda projects. Looking at https://pixi.sh/latest/advanced/pyproject_toml/ Pixi allows a combinations of PyPI dependencies and Conda dependencies to be declared in pyproject.toml:

AFAIUI, PyPI dependencies are declared in the project.dependencies section of pyproject.toml, and seems to obey PEP621. (This part should thus already be supported by FawltyDeps.)

Next, Conda dependencies are declared in a [tool.pixi.dependencies] section. In the case of overlaps (a dependency name that occurs in both sections), the Conda dependency is preferred and the PyPI dependency is ignored.

The dependencies specifiers in the [tool.pixi.dependencies] section initially appear to be formatted in the same as PEP621's project.dependencies (see https://packaging.python.org/en/latest/specifications/dependency-specifiers/ for details), but according to https://pixi.sh/latest/reference/project_configuration/#the-dependencies-tables there are slight differences in these formats:

[Conda] Dependencies are defined using a VersionSpec. [...] [PyPI] dependencies don't follow the conda matchspec specification. The version is a string specification of the version according to PEP404/PyPA. Additionally, a list of extra's can be included, which are essentially optional dependencies. Note that this version is distinct from the conda MatchSpec type.

It is not yet clear to me whether the conda version specifiers are a subset of the PyPI version specifiers (and can thus reuse the same parser), or if a separate parser is necessary.

What does seem clear is that the "pinned" version specifiers that specify an exact package build elsewhere in the Conda ecosystem (e.g. the requests=2.32.3=py38h06a4308_0 string that would appear in a conda-generated environment.yml or requirements.txt) are NOT allowed by Pixi here.


Note that Pixi (like Conda) also supports non-Python projects (which is outside FawltyDeps' scope), and also have their own pixi.toml file used (instead of pyproject.toml) to configure Pixi projects. Supporting pixi.toml in FawltyDeps might be a valuable feature at some point, but is outside the scope of this issue.

jherland commented 2 months ago

PR #455 adds support for parsing the various Pixi fields in pyproject.toml.

Please note that this does not introduce any differentiation between Conda dependencies and PyPI dependencies, meaning that we for now proceed on the flawed assumption that a Conda dependency named FOO will map one-to-one to a Python package named FOO. This is certainly not true for Conda dependencies that are not Python packages, and it probably isn't even true for all Conda dependencies that do include Python packages.

The PR also bypasses the issue of how to parse Conda dependency version specifiers, merely by not parsing them at all. FalwtyDeps has so far not cared about versions of dependencies anyway, and hopefully we can get away with the same for Conda dependencies

Still, this is at least one way to dip our toe into the Conda waters. Next, we can add support for parsing pixi.toml and then environment.yml. And we can then revisit the flawed conda==python assumption in later PRs.