search

aboutcode-org / python-inspector

Inspect Python code and PyPI package manifests. Resolve Python dependencies.
22 stars 19 forks source link

Problem with resolution #101

Open pombredanne opened 2 years ago

pombredanne commented 2 years ago

There are a couple issues: 1. this file is not a valid requirements file IMHO and we do not return any error: I save this in re.txt and ran: python-inspector --json - -p 3.10 -o linux --version -r re.txt

numpy>=1.0
scipy
xobjects
xpart
xdeps

[tests]
cpymad
PyHEADTAIL
pytest

This was copied by mistake as is from the setup.py of xtrack-0.23.2.tar.gz from https://pypi.org/project/xtrack/#files

2. trying to inspect it yields this error which is not right becuase this setup.py is perfectly parsable:

>>> resolver_api(requirement_files=["re.txt"], python_version="310", operating_system="linux")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/api.py", line 254, in resolve_dependencies
    resolution, purls = resolve(
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/api.py", line 313, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/api.py", line 351, in get_resolved_dependencies
    resolver_results = resolver.resolve(requirements=requirements, max_rounds=max_rounds)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 521, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 402, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 238, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 228, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 600, in get_dependencies
    return list(self._iter_dependencies(candidate))
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 591, in _iter_dependencies
    for r in self.get_requirements_for_package(purl=purl, candidate=candidate):
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 470, in get_requirements_for_package_from_pypi_simple
    yield from get_requirements_from_python_manifest(
  File "/tmp/scancode-toolkit/venv/lib/python3.8/site-packages/python_inspector/resolution.py", line 304, in get_requirements_from_python_manifest
    raise Exception(
Exception: Unable to collect setup.py dependencies securely: .cache/thirdparty/extracted_sdists/PyHEADTAIL-1.16.0/PyHEADTAIL-1.16.0/setup.py

3. Trying to call the API with the setup.py is not right: See https://files.pythonhosted.org/packages/7c/dd/80d8ec272179941bf6cc181446127da194eb82b7138549a6939583c4c318/PyHEADTAIL-1.16.0.tar.gz (which should have been in the graph above) returns an imcomplete deps tree (or rather nothing)

>>> resolver_api(setup_py_file=".cache/thirdparty/extracted_sdists/PyHEADTAIL-1.16.0/PyHEADTAIL-1.16.0/setup.py", python_version="310", operating_system="linux")
Resolution(resolution=[], packages=[], files=[{'type': 'file', 'path': '.cache/thirdparty/extracted_sdists/PyHEADTAIL-1.16.0/PyHEADTAIL-1.16.0/setup.py', 'package_data': [{'type': 'pypi', 'namespace': None, 'name': 'PyHEADTAIL', 'version': '1.16.0', 'qualifiers': {}, 'subpath': None, 'primary_language': 'Python', 'description': 'CERN PyHEADTAIL numerical n-body simulation code for simulating macro-particle beam dynamics with collective effects.', 'release_date': None, 'parties': [{'type': 'person', 'role': 'author', 'name': 'Kevin Li', 'email': 'Kevin.Shing.Bruce.Li@cern.ch', 'url': None}, {'type': 'person', 'role': 'maintainer', 'name': 'Adrian Oeftiger', 'email': 'Adrian.Oeftiger@cern.ch', 'url': None}], 'keywords': [], 'homepage_url': 'https://github.com/PyCOMPLETE/PyHEADTAIL', 'download_url': None, 'size': None, 'sha1': None, 'md5': None, 'sha256': None, 'sha512': None, 'bug_tracking_url': None, 'code_view_url': None, 'vcs_url': None, 'copyright': None, 'license_expression': None, 'declared_license': {}, 'notice_text': None, 'source_packages': [], 'file_references': [], 'extra_data': {}, 'dependencies': [{'purl': 'pkg:pypi/h5py', 'extracted_requirement': 'h5py', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}, {'purl': 'pkg:pypi/numpy', 'extracted_requirement': 'numpy', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}, {'purl': 'pkg:pypi/scipy', 'extracted_requirement': 'scipy', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}, {'purl': 'pkg:pypi/cython', 'extracted_requirement': 'cython', 'scope': 'setup', 'is_runtime': True, 'is_optional': False, 'is_resolved': False, 'resolved_package': {}, 'extra_data': {}}], 'repository_homepage_url': 'https://pypi.org/project/PyHEADTAIL', 'repository_download_url': 'https://pypi.org/packages/source/P/PyHEADTAIL/PyHEADTAIL-1.16.0.tar.gz', 'api_data_url': 'https://pypi.org/pypi/PyHEADTAIL/1.16.0/json', 'datasource_id': 'pypi_setup_py', 'purl': 'pkg:pypi/pyheadtail@1.16.0'}]}])

4. Scanning that seame setup.py with SCTK works without a glitch with these deps

                dependencies:
                    -   purl: pkg:pypi/h5py
                        extracted_requirement: h5py
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}
                    -   purl: pkg:pypi/numpy
                        extracted_requirement: numpy
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}
                    -   purl: pkg:pypi/scipy
                        extracted_requirement: scipy
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}
                    -   purl: pkg:pypi/cython
                        extracted_requirement: cython
                        scope: setup
                        is_runtime: yes
                        is_optional: no
                        is_resolved: no
                        resolved_package: {}
                        extra_data: {}

5. running this does not have the extra requires: python-inspector --spec "xtrack==0.23.3" --json - -p 3.10 -o linux

"resolved_dependencies_graph": [
    {
      "package": "pkg:pypi/cffi@1.15.1",
      "dependencies": [
        "pkg:pypi/pycparser@2.21"
      ]
    },
    {
      "package": "pkg:pypi/lark@1.1.4",
      "dependencies": []
    },
    {
      "package": "pkg:pypi/numpy@1.23.5",
      "dependencies": []
    },
    {
      "package": "pkg:pypi/pycparser@2.21",
      "dependencies": []
    },
    {
      "package": "pkg:pypi/scipy@1.9.3",
      "dependencies": [
        "pkg:pypi/numpy@1.23.5"
      ]
    },
    {
      "package": "pkg:pypi/xdeps@0.0.10",
      "dependencies": [
        "pkg:pypi/lark@1.1.4"
      ]
    },
    {
      "package": "pkg:pypi/xobjects@0.1.27",
      "dependencies": [
        "pkg:pypi/cffi@1.15.1",
        "pkg:pypi/numpy@1.23.5"
      ]
    },
    {
      "package": "pkg:pypi/xpart@0.10.0",
      "dependencies": [
        "pkg:pypi/numpy@1.23.5",
        "pkg:pypi/scipy@1.9.3",
        "pkg:pypi/xobjects@0.1.27"
      ]
    },
    {
      "package": "pkg:pypi/xtrack@0.23.3",
      "dependencies": [
        "pkg:pypi/numpy@1.23.5",
        "pkg:pypi/scipy@1.9.3",
        "pkg:pypi/xdeps@0.0.10",
        "pkg:pypi/xobjects@0.1.27",
        "pkg:pypi/xpart@0.10.0"
      ]
    }
  ]

6. we should have a way to return partial resolutions and we should not fail silently