Extract dependencies from pyproject.toml

[mknorps]

Add function to extract dependencies from pyproject.toml.

Note, consider development and library dependencies separately.

[mknorps]

How dependencies are declared in pyproject.toml?

Poetry

https://python-poetry.org/docs/managing-dependencies/

To declare a new dependency group, use a tool.poetry.group.<group> section where <group> is the name of your dependency group (for instance, test):

[tool.poetry.group.test.dependencies]
pytest = "^6.0.0"
pytest-mock = "*"

The dependencies declared in tool.poetry.dependencies are part of an implicit main group.

Poetry also supports "extras" with following syntax:

[tool.poetry.extras]
mysql = ["mysqlclient"]
pgsql = ["psycopg2"]
databases = ["mysqlclient", "psycopg2"]

https://python-poetry.org/docs/pyproject#extras

[mknorps]

General pyproject.toml

As stated in PEP 621:

Format: Array of PEP 508 strings (dependencies) and a table with values of arrays of PEP 508 strings (optional-dependencies) Core metadata: Requires-Dist and Provides-Extra (link, link) Synonyms

• Flit: requires for required dependencies, requires-extra for optional dependencies (link)
• Poetry: [tool.poetry.dependencies] for dependencies (both required and for development), [tool.poetry.extras] for optional dependencies (link)
• Setuptools: install_requires for required dependencies, extras_require for optional dependencies (link) The (optional) dependencies of the project.

For dependencies, it is a key whose value is an array of strings. Each string represents a dependency of the project and MUST be formatted as a valid PEP 508 string. Each string maps directly to a Requires-Dist entry in the core metadata.

For optional-dependencies, it is a table where each key specifies an extra and whose value is an array of strings. The strings of the arrays must be valid PEP 508 strings. The keys MUST be valid values for the Provides-Extra core metadata. Each value in the array thus becomes a corresponding Requires-Dist entry for the matching Provides-Extra metadata.

Example:

[project]
name = "spam"
...
dependencies = [
  "httpx",
  "gidgethub[httpx]>4.0.0",
  "django>2.1; os_name != 'nt'",
  "django>2.0; os_name == 'nt'"
]

[project.optional-dependencies]
test = [
  "pytest < 5.0.0",
  "pytest-cov[all]"
]

[mknorps]

Parsing toml is available in the standard library from version 3.11 (see https://peps.python.org/pep-0680/, https://docs.python.org/3/library/tomllib.html). As FawltyDeps should support older version of Python we use tomli as std library implementation is based on this.

[Nour-Mws]

Recap-ing the information posted by @mknorps above:

Native pyproject.toml

Section	Format	Example
dependencies (under [project])	Array of PEP 508 strings (dependencies)	[project] name = "spam" ... dependencies = [ "httpx", "gidgethub[httpx]>4.0.0"]
[project.optional-dependencies]	table with values of arrays of PEP 508 strings	[project.optional-dependencies] test = [ "pytest < 5.0.0", "pytest-cov[all]" ]

Poetry

Section	Format	Example
[tool.poetry.dependencies]	PEP 508 strings (dependencies)	[tool.poetry.dependencies] python = "^3.7.2" isort = "^5.10"
[tool.poetry.group.<group-name>.dependencies]	PEP 508 strings (dependencies)	[tool.poetry.group.dev.dependencies] black = "^22" mypy = "^0.991"
[tool.poetry.extras]	table with values of arrays of PEP 508 strings	[tool.poetry.extras] mysql = ["mysqlclient"] databases = ["mysqlclient", "psycopg2"]

[Nour-Mws]

I suggest only parsing native pyproject.toml fields for PR #34 and considering parsing Poetry related fields in a subsequent PR. I think supporting Poetry-specific fields in pyproject.toml is a necessity given the popularity of the library. But are there any similar libraries that we know of that we would then need to consider as well?

A bit off topic, but Poetry is planning to support PEP 621-styled pyproject.toml in the next version 2.0 as discussed in https://github.com/python-poetry/roadmap/issues/3. See also https://github.com/python-poetry/poetry/discussions/5833.

[mknorps]

I suggest only parsing native pyproject.taml fields for PR #34 and considering parsing Poetry related fields in a subsequent PR.

Is this still valid? #34 has already both.

I think supporting Poetry-specific fields in pyproject.toml is a necessity given the popularity of the library. But are there any similar libraries that we know of that we would then need to consider as well?

Yes, in https://github.com/tweag/FawltyDeps/issues/25#issuecomment-1369009027, PEP621 lists also flit and setuptools. It has to be checked if the dependencies definitions are different there.

A bit off topic, but Poetry is planning to support PEP 621-styled pyproject.toml in the next version 2.0 as discussed in python-poetry/roadmap#3. See also python-poetry/poetry#5833.

:tada:

[Nour-Mws]

Is this still valid? https://github.com/tweag/FawltyDeps/pull/34 has already both.

Nope. I went ahead and parsed both Poetry and the PEP 621 style.

In https://github.com/tweag/FawltyDeps/issues/25#issuecomment-1369009027, PEP621 lists also flit and setuptools. It has to be checked if the dependencies definitions are different there.

• The new metadata style in Flit seems to now support the same pyproject.toml dependency fields as PEP 621. The Flit-specific fields (requires and requires-extra) are part of the old style metadata. I opened issue #49 to track parsing dependencies from those fields.
• I opened issue #50 to track parsing setuptools-specific dependency fields in pyproject.toml
• There are other "build backends" supporting PEP 621 and I'm not sure on the difference between a build backend and a build tool. PEP 621 only lists the 3 build tools that you quoted in your comment above. Issue #51 now tracks this.