PyPI parser does not handle local version identifiers

[furi0us333]

Overview

We have a customer using local forked versions of some packages and they are getting errors. This seems due to the parser not handling the local version identifier.

Reference: https://packaging.python.org/en/latest/specifications/version-specifiers/#local-version-identifiers

How To Reproduce

Steps to reproduce this behavior:

1. Create a requirements.txt file
2. Populate it with a sample package that uses the local version identifier scheme (e.g., pyyaml==5.3.1+Local.version1)
3. Run phylum parse on the file
4. See error

Expected Behavior

These packages won't be found in the public registry, so we should probably just skip over them.

Additional Context

This is currently blocking a customer and should be addressed as soon as possible.

[maxrake]

Modifying the test fixture to include the example:

❯ git diff tests/fixtures/requirements-locked.txt
diff --git a/tests/fixtures/requirements-locked.txt b/tests/fixtures/requirements-locked.txt
index 07297db..4c42283 100644
--- a/tests/fixtures/requirements-locked.txt
+++ b/tests/fixtures/requirements-locked.txt
@@ -39,3 +39,5 @@ other-registry-a==3.2.1
 -ihttps://mirror2.phylum.io/simple/
 --extra-index-url=https://mirror3.phylum.io/simple/
 other-registry==1.2.3
+
+pyyaml==5.3.1+Local.version1

...results in test failures like this:

Details (click to expand...)

```bash ❯ cargo test -p phylum_lockfile --lib Compiling phylum_lockfile v0.1.0 (/Users/maxrake/dev/phylum/localdev/cli/lockfile) Finished `test` profile [unoptimized + debuginfo] target(s) in 2.27s Running unittests src/lib.rs (target/debug/deps/phylum_lockfile-a0065491f45e196b) running 62 tests test csharp::tests::test_overlapped_case ... ok test csharp::tests::lock_parse_csproj ... ok test csharp::tests::packages_lock ... ok test cyclonedx::tests::test_if_lockfile ... ok test cyclonedx::tests::parse_cyclonedx_nested_components ... ok test cyclonedx::tests::test_ignore_unsupported_ecosystem ... ok test golang::tests::parse_go_mod_unsupported ... ok test golang::tests::parse_go_mod ... ok test cargo::tests::parse_cargo_lock_v2 ... ok test csharp::tests::strip_utf8_bom ... ok test java::tests::lock_parse_gradle ... ok test java::tests::lock_parse_effective_pom_cp1252 ... ok test java::tests::lock_parse_effective_pom ... ok test javascript::tests::empty_pnpm ... ok test javascript::tests::empty_yarn_v1 ... ok test javascript::tests::empty_yarn_v2 ... ok test javascript::tests::lock_parse_package ... ok test javascript::tests::lock_parse_package_v7 ... ok test javascript::tests::lock_parse_yarn_v1_malformed_fails - should panic ... ok test javascript::tests::lock_parse_yarn_v1_simple ... ok test javascript::tests::lock_parse_yarn_v1 ... ok test cyclonedx::tests::parse_cyclonedx_1_3 ... ok test cyclonedx::tests::parse_cyclonedx_1_5 ... ok test cyclonedx::tests::parse_cyclonedx_1_4 ... ok test python::tests::fails_loose_requirements ... ok test python::tests::lock_parse_pipfile ... ok test javascript::tests::pnpm ... ok test python::tests::parse_pipfile ... ok test javascript::tests::pnpm_v9 ... ok test python::tests::parse_requirements ... FAILED test ruby::tests::empty_lockfile ... ok test python::tests::parse_poetry_lock_v2 ... ok test ruby::tests::lock_parse_gem ... ok test spdx::tests::parse_spdx_2_2_json ... ok test javascript::tests::lock_parse_yarn ... ok test spdx::tests::pkg_locator ... ok test spdx::tests::pkg_locator_tag_value ... ok test spdx::tests::parse_spdx_2_3_yaml ... ok test spdx::tests::removes_self_identified_package ... ok test cargo::tests::parse_cargo_lock_v1 ... ok test spdx::tests::spdx_2_3_unsupported_ecosystem ... ok test spdx::tests::tag_value_ecosystem_from_download_location ... ok test spdx::tests::tag_value_fail_missing_version ... ok test spdx::tests::tag_value_unsupported_ecosystem ... ok test spdx::tests::test_if_lockfile ... ok test spdx::tests::tag_value_ignore_unknown_ecosystem ... ok test tests::get_path_parser_identifies_lockfile_parsers ... ok test tests::lockfile_format_display_serializes_correctly ... ok test spdx::tests::test_missing_version ... ok test tests::lockfile_format_from_str_parses_correctly ... ok test tests::lockfile_format_name_matches_to_string ... ok test python::tests::parse_poetry_lock_v1 ... ok test tests::no_duplicate_requirements ... ok test tests::setup_without_pyproject ... ok test tests::skip_setup_with_pyproject ... ok test golang::tests::parse_go_sum ... ok test cargo::tests::parse_cargo_lock_v3 ... ok test java::tests::lock_parse_workspace_effective_pom ... ok test spdx::tests::parse_spdx_2_3_json ... ok test spdx::tests::parse_spdx_2_2_tag_value ... ok test parse_depfile::tests::it_can_identify_lock_file_types ... FAILED test tests::parsers_only_parse_their_lockfiles ... ok failures: ---- python::tests::parse_requirements stdout ---- thread 'python::tests::parse_requirements' panicked at lockfile/src/python.rs:240:14: called `Result::unwrap()` on an `Err` value: Failed to parse requirements file Caused by: 0: at line 43, in Eof: pyyaml==5.3.1+Local.version1 ^ ---- parse_depfile::tests::it_can_identify_lock_file_types stdout ---- thread 'parse_depfile::tests::it_can_identify_lock_file_types' panicked at lockfile/src/parse_depfile.rs:244:60: called `Result::unwrap()` on an `Err` value: UnknownManifestFormat("../tests/fixtures/requirements-locked.txt") failures: parse_depfile::tests::it_can_identify_lock_file_types python::tests::parse_requirements test result: FAILED. 60 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.88s error: test failed, to rerun pass `-p phylum_lockfile --lib` ```

[maxrake]

Attempting to parse a lockfile containing an entry like the example provided results in the parser thinking the file is a manifest instead. Then, the lockfile generation fails due to the package version not existing on PyPI...which makes sense because the spec (which comes from the historical PEP 440) does state:

As the Python Package Index is intended solely for indexing and hosting upstream projects, it MUST NOT allow the use of local version identifiers.

This further reinforces the suggestion that lockfile entries with local version identifiers should be skipped so as not to affect processing the remainder of the file.

❯ phylum --version
phylum v6.3.0

❯ cat requirements.txt
phylum==0.44.0
pyyaml==5.3.1+Local.version1

❯ phylum parse -t pip requirements.txt
Generating lockfile for manifest "requirements.txt" using Pip…
❗ Error: Lockfile generation failed! For details, see: https://docs.phylum.io/cli/lockfile_generation

Caused by:
    package manager quit unexpectedly (code: Some(1)):

    WARNING: The directory '/Users/maxrake/Library/Caches/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
    ERROR: Ignored the following versions that require a different python version: 0.18.0 Requires-Python >=3.7,<3.12; 0.19.0 Requires-Python >=3.7,<3.12; 0.20.0 Requires-Python >=3.7,<3.12; 0.21.0 Requires-Python >=3.7,<3.12; 0.22.0 Requires-Python >=3.7,<3.12; 0.22.1 Requires-Python >=3.7,<3.12; 0.23.0 Requires-Python >=3.7,<3.12; 0.23.1 Requires-Python >=3.7,<3.12; 0.24.0 Requires-Python >=3.7,<3.12; 0.24.1 Requires-Python >=3.7,<3.12; 0.25.0 Requires-Python >=3.7,<3.12; 0.26.0 Requires-Python >=3.8,<3.12; 0.27.0 Requires-Python >=3.8,<3.12; 0.28.0 Requires-Python >=3.8,<3.12; 0.28.1 Requires-Python >=3.8,<3.12; 0.29.0 Requires-Python >=3.8,<3.12; 0.30.0 Requires-Python >=3.8,<3.12; 0.30.1 Requires-Python >=3.8,<3.12; 0.31.0 Requires-Python >=3.8,<3.12; 0.32.0 Requires-Python >=3.8,<3.12; 0.32.1 Requires-Python >=3.8,<3.12; 0.33.0 Requires-Python >=3.8,<3.12; 0.34.0 Requires-Python >=3.8,<3.12; 0.35.0 Requires-Python >=3.8,<3.12; 0.35.1 Requires-Python >=3.8,<3.12; 0.35.2 Requires-Python >=3.8,<3.12; 0.36.0 Requires-Python >=3.8,<3.12
    ERROR: Could not find a version that satisfies the requirement pyyaml==5.3.1+Local.version1 (from versions: 3.10, 3.11, 3.12, 3.13b1, 3.13rc1, 3.13, 4.2b1, 4.2b2, 4.2b4, 5.1b1, 5.1b3, 5.1b5, 5.1, 5.1.1, 5.1.2, 5.2b1, 5.2, 5.3b1, 5.3, 5.3.1, 5.4b1, 5.4b2, 5.4, 5.4.1, 6.0b1, 6.0, 6.0.1)
    ERROR: No matching distribution found for pyyaml==5.3.1+Local.version1

❗ Error: Could not parse dependency file "requirements.txt" as "pip" type

Caused by:
    Dependency file parsing failed