search

aboutcode-org / python-inspector

Inspect Python code and PyPI package manifests. Resolve Python dependencies.
22 stars 19 forks source link

Analysis with 0.9.7 fails with `cannot import name 'Feature' from 'setuptools'` #133

Open MarcelBochtler opened 1 year ago

MarcelBochtler commented 1 year ago

Running the python-inspector on this requirements.txt, fails with the following error.

The same project can successfully be analyzed using python-inspector 0.9.6.

The command I used is:

python-inspector --python-version 310 --operating-system linux --json-pdt result.json --analyze-setup-py-insecurely --requirement requirements.txt --verbose
Warning: ambiguous setup provider in /Volumes/src/oss/ort/plugins/package-managers/python/src/funTest/assets/projects/synthetic/python-inspector/.cache/python_inspector/extracted_sdists/itsdangerous-0.24/itsdangerous-0.24/setup.py: candidates are ['setuptools', 'distutils.core']defaulting to 'distutils.core'.
Traceback (most recent call last):
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/resolve_cli.py", line 247, in resolve_dependencies
    resolution_result: Dict = resolver_api(
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/api.py", line 263, in resolve_dependencies
    resolution, purls = resolve(
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/api.py", line 322, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/api.py", line 360, in get_resolved_dependencies
    resolver_results = resolver.resolve(requirements=requirements, max_rounds=max_rounds)
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/resolvelib/resolvers.py", line 229, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/resolution.py", line 644, in get_dependencies
    return list(self._iter_dependencies(candidate))
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/resolution.py", line 635, in _iter_dependencies
    for r in self.get_requirements_for_package(purl=purl, candidate=candidate):
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/resolution.py", line 497, in get_requirements_for_package_from_pypi_simple
    yield from get_reqs_insecurely(
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/resolution.py", line 277, in get_reqs_insecurely
    yield from parse_reqs_from_setup_py_insecurely(setup_py=setup_py_location)
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/resolution.py", line 128, in parse_reqs_from_setup_py_insecurely
    for req in iter_requirements(level="", extras=[], setup_file=setup_py):
  File "/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/python_inspector/setup_py_live_eval.py", line 117, in iter_requirements
    exec(file_contents, g)
  File "<string>", line 6, in <module>
ImportError: cannot import name 'Feature' from 'setuptools' (/Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/setuptools/__init__.py)
pombredanne commented 1 year ago

@MarcelBochtler Thank you for the report. That's an odd one for sure. Can you tell which version of setuptools may be installed locally, e.g., in the context of python-inspector?

pombredanne commented 1 year ago

For reference, see also https://github.com/pypa/setuptools/issues/2423

pombredanne commented 1 year ago

@MarcelBochtler I see you are running Python 3.9, but the PR you mentioned above references 3.10 at https://github.com/oss-review-toolkit/ort/pull/7059/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R129

So if you run this what do you get?: ls -ald /Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/setup*

Also are you running on macOS or Linux?

MarcelBochtler commented 1 year ago

@MarcelBochtler I see you are running Python 3.9, but the PR you mentioned above references 3.10 at https://github.com/oss-review-toolkit/ort/pull/7059/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R129

I'm getting the same result using:

python-inspector --python-version 39 --operating-system macos --json-pdt result.json --analyze-setup-py-insecurely --requirement requirements.txt --verbose

So if you run this what do you get?: ls -ald /Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/setup*

❯ ls -ald /Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/setup*
drwx------ 41 bom1wa3 staff 1312 Jul 19  2021 /Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/setuptools
drwx------ 12 bom1wa3 staff  384 Jul 19  2021 /Users/bom1wa3/Library/Python/3.9/lib/python/site-packages/setuptools-50.3.2.dist-info

Also are you running on macOS or Linux?

Above test was done on macOS, but I'm getting the same issue in ORT's integration tests for the python inspector, which run on Linux: https://github.com/oss-review-toolkit/ort/actions/runs/5118814596/jobs/9203276843?pr=7059

fviernau commented 1 year ago

I've just reproduced this in ORT's docker image with the 0.10.0 release of python inspector. Note, that without --analyze-setup-py-insecurely the issue does not reproduce. It also does not reproduce when using MarkupSafe==1.0.1 instead of MarkupSafe==1.0.0.

requirements.txt:

MarkupSafe==1.0.0

reproducer:

python-inspector --python-version 311 --operating-system linux --json-pdt x.json  --requirement requirements.txt --analyze-setup-py-insecurely
Warning: ambiguous setup provider in /workspace/plugins/package-managers/python/src/funTest/assets/projects/synthetic/python-inspector/.cache/python_inspector/extracted_sdists/itsdangerous-0.24/itsdangerous-0.24/setup.py: candidates are ['setuptools', 'distutils.core']defaulting to 'distutils.core'.
/opt/python/versions/3.11.5/lib/python3.11/site-packages/setuptools/dist.py:498: SetuptoolsDeprecationWarning: Invalid dash-separated options
!!

        ********************************************************************************
        Usage of dash-separated 'upload-dir' will not be supported in future
        versions. Please use the underscore name 'upload_dir' instead.

        This deprecation is overdue, please update your project and remove deprecated
        calls to avoid build errors in the future.

        See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
        ********************************************************************************

!!
  opt = self.warn_dash_deprecation(opt, section)
Traceback (most recent call last):
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolve_cli.py", line 247, in resolve_dependencies
    resolution_result: Dict = resolver_api(
                              ^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/api.py", line 263, in resolve_dependencies
    resolution, purls = resolve(
                        ^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/api.py", line 322, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/api.py", line 360, in get_resolved_dependencies
    resolver_results = resolver.resolve(requirements=requirements, max_rounds=max_rounds)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/resolvelib/resolvers.py", line 229, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 644, in get_dependencies
    return list(self._iter_dependencies(candidate))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 635, in _iter_dependencies
    for r in self.get_requirements_for_package(purl=purl, candidate=candidate):
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 497, in get_requirements_for_package_from_pypi_simple
    yield from get_reqs_insecurely(
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 277, in get_reqs_insecurely
    yield from parse_reqs_from_setup_py_insecurely(setup_py=setup_py_location)
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 128, in parse_reqs_from_setup_py_insecurely
    for req in iter_requirements(level="", extras=[], setup_file=setup_py):
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/setup_py_live_eval.py", line 117, in iter_requirements
    exec(file_contents, g)
  File "<string>", line 6, in <module>
ImportError: cannot import name 'Feature' from 'setuptools' (/opt/python/versions/3.11.5/lib/python3.11/site-packages/setuptools/__init__.py)
ls -ald /opt/python/versions/3.11.5/lib/python3.11/site-packages/setup*
drwxr-xr-x 8 ort ort 4096 Oct 31 23:08 /opt/python/versions/3.11.5/lib/python3.11/site-packages/setuptools
drwxr-xr-x 2 ort ort 4096 Oct 31 23:08 /opt/python/versions/3.11.5/lib/python3.11/site-packages/setuptools-68.2.2.dist-info