search

oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.57k stars 308 forks source link

Python 3.11 Support #7476

Closed mawl closed 11 months ago

mawl commented 1 year ago

Hi there,

As a follow-up to https://github.com/oss-review-toolkit/ort/issues/7333:

How can I teach poetry / pip / python-inspector to use python version 3.11 instead of 3.10?

https://github.com/oss-review-toolkit/ort/issues/5740 says, that this should work:

.ort.yml

analyzer:
  package_managers:
    PIP:
      options:
        operatingSystem: 'linux'
        pythonVersion: '3.11'

I have build your Dockerfile with actual python components:

      --build-arg PYTHON_VERSION=3.11.4
      --build-arg PYENV_GIT_TAG=v2.3.23
      --build-arg PIPTOOL_VERSION=23.2.1      
      --build-arg PYTHON_POETRY_VERSION=1.5.1

pyenv versions says: * 3.11.4 (set by /opt/python/version)

But this get's executed and the dependency tree stays empty:

python-inspector --python-version 310 --operating-system linux ...

Full log:

13:24:30.432 [main] INFO  org.ossreviewtoolkit.analyzer.Analyzer - Starting Poetry analysis.
13:24:30.438 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.analyzer.PackageManager - Using Poetry to resolve dependencies for path 'test/python/poetry/poetry.lock'...
13:24:30.448 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.plugins.packagemanagers.python.Poetry - Generating 'requirements-from-poetry.txt' file in '/builds/company/compliance/license-scanning/test/python/poetry' directory...
13:24:30.455 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.EnvironmentVariableFilter - Filtering out these variables from the environment: [...
13:24:30.459 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'poetry export --without-hashes --format=requirements.txt' in '/builds/company/compliance/license-scanning/test/python/poetry'...
13:24:30.997 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - dnspython==2.4.1 ; python_version >= "3.11" and python_version < "3.12"
13:24:30.998 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - pendulum==2.1.2 ; python_version >= "3.11" and python_version < "3.12"
13:24:30.998 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - pymongo==4.4.1 ; python_version >= "3.11" and python_version < "3.12"
13:24:30.998 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - python-dateutil==2.8.2 ; python_version >= "3.11" and python_version < "3.12"
13:24:30.998 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - pytzdata==2020.1 ; python_version >= "3.11" and python_version < "3.12"
13:24:30.998 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - six==1.16.0 ; python_version >= "3.11" and python_version < "3.12"
13:24:30.998 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - 
13:24:31.009 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.plugins.packagemanagers.python.Pip - Resolving dependencies for '/builds/company/compliance/license-scanning/test/python/poetry/requirements-from-poetry.txt' with Python version '3.10' and operating system 'linux'.
13:24:31.011 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.EnvironmentVariableFilter - Filtering out these variables from the environment: [...
13:24:31.013 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'python-inspector --python-version 310 --operating-system linux --json-pdt /tmp/ort-PythonInspector14862222632561284951/python-inspector7097629943133964626.json --analyze-setup-py-insecurely --requirement /builds/company/compliance/license-scanning/test/python/poetry/requirements-from-poetry.txt --verbose' in '/builds/company/compliance/license-scanning/test/python/poetry'...
13:24:31.864 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - Resolving dependencies...
13:24:31.864 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - Using netrc file /home/ort/.netrc
13:24:31.864 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - direct_dependencies:
13:24:31.864 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -  DependentPackage(purl='pkg:pypi/dnspython@2.4.1', extracted_requirement='dnspython==2.4.1; python_version >= "3.11" and python_version < "3.12"', scope='install')
13:24:31.864 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -  DependentPackage(purl='pkg:pypi/pendulum@2.1.2', extracted_requirement='pendulum==2.1.2; python_version >= "3.11" and python_version < "3.12"', scope='install')
13:24:31.864 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -  DependentPackage(purl='pkg:pypi/pymongo@4.4.1', extracted_requirement='pymongo==4.4.1; python_version >= "3.11" and python_version < "3.12"', scope='install')
13:24:31.864 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -  DependentPackage(purl='pkg:pypi/python-dateutil@2.8.2', extracted_requirement='python-dateutil==2.8.2; python_version >= "3.11" and python_version < "3.12"', scope='install')
13:24:31.864 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -  DependentPackage(purl='pkg:pypi/pytzdata@2020.1', extracted_requirement='pytzdata==2020.1; python_version >= "3.11" and python_version < "3.12"', scope='install')
13:24:31.864 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -  DependentPackage(purl='pkg:pypi/six@1.16.0', extracted_requirement='six==1.16.0; python_version >= "3.11" and python_version < "3.12"', scope='install')
13:24:31.865 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - environment: Environment(python_version='310', operating_system='linux')
13:24:31.865 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - repos:
13:24:31.865 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -  PypiSimpleRepository(index_url='https://pypi.org/simple', credentials=None)
13:24:31.865 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - done!
13:24:31.865 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -

Thanks for your help.

fviernau commented 11 months ago

I've also investigated this a bit. I found that

  1. Specifying a python version (range) in pyproject.toml is mandatory
  2. The poetry export ... command used by Poetry.kt to generate a `requirements.txt also exports the version constraints for the python version.

Once ORT calls python-inspector to analyze the generated requirements.txt file with a python version (-p option) not in the python version range specified in pyproject.toml then the result would always be empty. So, I wonder if we should change the logic such that Poetry.kt selects an arbitrary python version from available python versions matching the range, in case no version has been specified.

Looking for feedback on this...