Parse error if comment # starts in the middle of the line

[wkl3nk]

Version: 0.10.0, 0.12.0 To reproduce, the following repository can be used for scanning: https://github.com/danielhrisca/asammdf

This contains a dependency to numexpr that itself has a line in its requirement.txt has has a comment that starts in the middle of the line: numpy >= 1.23.0 # keep in sync with NPY_TARGET_VERSION (setup.py)

Here is the output of a scan with python-inspector:

asammdf % python-inspector --python-version 311 --operating-system linux --json-pdt /tmp/my-dependencies.json --analyze-setup-py-insecurely --requirement requirements.txt --verbose

Resolving dependencies...
direct_dependencies:
 DependentPackage(purl='pkg:pypi/canmatrix', extracted_requirement='canmatrix[arxml,dbc]>=1.0', scope='install')
 DependentPackage(purl='pkg:pypi/lz4', extracted_requirement='lz4', scope='install')
 DependentPackage(purl='pkg:pypi/numexpr', extracted_requirement='numexpr', scope='install')
 DependentPackage(purl='pkg:pypi/numpy', extracted_requirement='numpy>=1.23.0', scope='install')
 DependentPackage(purl='pkg:pypi/pandas', extracted_requirement='pandas', scope='install')
 DependentPackage(purl='pkg:pypi/typing-extensions', extracted_requirement='typing_extensions', scope='install')
 DependentPackage(purl='pkg:pypi/python-dateutil', extracted_requirement='python-dateutil', scope='install')
 DependentPackage(purl='pkg:pypi/isal', extracted_requirement='isal; platform_machine == "x86_64" or platform_machine == "AMD64"', scope='install')
 DependentPackage(purl='pkg:pypi/lxml', extracted_requirement='lxml>=4.9.3', scope='install')
environment: Environment(python_version='311', operating_system='linux')
repos:
 PypiSimpleRepository(index_url='https://pypi.org/simple', credentials=None)
Traceback (most recent call last):
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/packvers/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/pyparsing/util.py", line 256, in _inner
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/pyparsing/core.py", line 1200, in parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected string_end, found '#'  (at char 16), (line:1, col:17)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolve_cli.py", line 257, in resolve_dependencies
    resolution_result: Dict = resolver_api(
                              ^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/api.py", line 280, in resolve_dependencies
    resolution, purls = resolve(
                        ^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/api.py", line 344, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/api.py", line 385, in get_resolved_dependencies
    resolver_results = resolver.resolve(requirements=requirements, max_rounds=max_rounds)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/resolvelib/resolvers.py", line 229, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 652, in get_dependencies
    return list(self._iter_dependencies(candidate))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 643, in _iter_dependencies
    for r in self.get_requirements_for_package(purl=purl, candidate=candidate):
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 502, in get_requirements_for_package_from_pypi_simple
    yield from get_reqs_insecurely(
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 277, in get_reqs_insecurely
    yield from parse_reqs_from_setup_py_insecurely(setup_py=setup_py_location)
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 128, in parse_reqs_from_setup_py_insecurely
    for req in iter_requirements(level="", extras=[], setup_file=setup_py):
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/setup_py_live_eval.py", line 151, in iter_requirements
    pkg = Requirement(req)
          ^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/packvers/requirements.py", line 104, in __init__
    raise InvalidRequirement(
packvers.requirements.InvalidRequirement: Parse error at "'# keep i'": Expected string_end

Is it allowed to have comments start in the middle of the line? I guess the Requirements File Format allows it when saying

A line that begins with # is treated as a comment and ignored. Whitespace followed by a # causes the # and the remainder of the line to be treated as a comment.

[sebassz]

This is most likely the same as mentioned in #152. There is already an open PR in the underlying packvers library to address this problem https://github.com/nexB/packvers/pull/3.