Handle single PEP-508 styled requirements

[Nour-Mws]

From https://github.com/tweag/FawltyDeps/pull/34#discussion_r1065626971:

We are starting to use parse_requirements_contents() from many different places now, and it seems most of them do not expect a full file contents a la requirements.txt, but rather expect a single (or multiple?) specifiers formatted according to PEP508. Suggest maybe (in a separate PR) to make a separate parse_pep508_specifier() that calls Requirement.parse(s) direcly?

[jherland]

Update after #227 was merged:

• PEP 508 is the "full" specification that describes the format of the dependencies that we might find in e.g. a requirements.txt file:

  • It encompasses complex use cases such as these examples:
  • requests [security,tests] >= 2.8.1, == 2.8.* ; python_version < "2.7"
  • pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
  • The only authoritative source/implementation for this is really pip itself, but they do not provide a stable API suitable for reuse.
  • The pip-requirements-parser project does provide a reusable package, and this is what we now use (as of #227)
  • There are several other project we could use, but pip-requirements-parser argues well for why it's the best choice.

• PEP 440 specifies a more limited format that includes version specifiers, but not the extra bells and whistles that PEP 508 includes.

The real questions are:

1. what kind of specifications do the various file formats use/need?
2. and how are we currently supporting these?

For 1, setuptools^1 specifies that pyproject.toml (following PEP 621), setup.cfg, and setup.py all need to comply with PEP 508 (and PEP 440) for their dependency specifications.

For 2, currently in our code:

• parse_one_req() is a helper that uses pkg_resources.Requirement.parse(), which claims to support PEP 508.
• parse_requirements_txt() (formerly known as parse_requirements_contents()) now uses pip-requirements-parser to parse full requirements.txt files.
• parse_setup_py() uses parse_one_req(). This is in line with what setuptools specifies^1.
• [ ] parse_setup_cfg() currently piggy-backs on parse_requirements_txt() via an ugly temp file hack. We should really examine whether this can be solved with parse_one_req() instead, as this should be sufficient, according to the what setuptools specifies^1.
• parse_pyproject_toml() uses parse_one_req(). This is in line with what setuptools specifies^1 for PEP 621 files, but Poetry does use some weird version specifiers (e.g. the ^ in mypy = "^0.991") which might be outside PEP 440.
  • [ ] We should confirm that these do not trip us up.

[mknorps]

@mknorps @jherland Check what is the status of this problem right now.

[obscurerichard]

Dealing with the full gamut of PEP-508 styled requirements is probably going to be a tar pit since the only full implementation of this is probably in pip internals that are not exposed with a stable and supported API. I've worked on practical projects (such as https://github.com/freezingsaddles/freezing-web for example) where the setup.py used to rely on pip internal libraries to parse requirements.txt but then broke when the pip internals changed.

So trying to get full support for this might be a lot of effort for relatively low reward.

[jherland]

Agree with @obscurerichard: we certainly don't want to dig further into pip internals than we absolutely have to.

With https://github.com/tweag/FawltyDeps/pull/445 we migrate parse_one_req() from depending on pkg_resources to depending on packaging.requirements instead. packaging.requirements claims to implement https://packaging.python.org/en/latest/specifications/dependency-specifiers/#dependency-specifiers which itself claims to supersede PEP 508. Given the deprecation of pkg_resources, this is clearly the better alternative AFAICS.

This reimplementation of parse_one_req() affects its callers: parse_setup_py() and parse_pyproject_toml(), but AFAICS there is no user-visible effect of this change.

parse_requirements_txt() remains on pip-requirements-parser as that provides a better API for parsing full files (rather than disparate specifiers), and the ugly temp file hack in parse_setup_cfg() also remains for now.

I suggest we either close this bug, or refocus it on the two remaining tasks mentioned above:

• [ ] Rewrite parse_setup_cfg() to use parse_one_req() instead of parse_requirements_txt()
• [ ] Verify that Poetry-style ^ version specifiers in pyproject.toml do not cause problems for parsing dependency specifiers.