search

oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.59k stars 309 forks source link

python-inspector specify platform support #5740

Closed pepper-jk closed 2 years ago

pepper-jk commented 2 years ago

Context:

We have some windows python tools, which are scanned with ORT for licensing purposes. Even though the tool is created for windows and scanned on linux, ORT collects the macOS artifacts of the dependencies:

2022-09-05 15 20 50-ORT Scan Report — Mozilla Firefox

Feature Request:

Allow us to specify python version and platform (macos, linux, windows) manually, so that we can avoid potential licensing errors.

Current Workaround:

python-inspector already allows users to specify the platform:

  -p, --python-version PYVER      Python version to use for dependency
                                  resolution.  [default: 38]
  -o, --operating-system OS       OS to use for dependency resolution.
                                  [default: linux]

I tried to hard code windows into my local ort image by editiing the run function in the PythonInspector object:

    fun run(
        workingDir: File,
        outputFile: String,
        definitionFile: File,
        pythonVersion: String = "38",
    ): ProcessCapture {
        val commandLineOptions = buildList {
            add("--python-version")
            add(pythonVersion)

            add("--operating-system")
            add("windows")

            add("--json-pdt")
            add(outputFile)

            if (definitionFile.name == "setup.py") {
                add("--setup-py")
            } else {
                add("--requirement")
            }

            add(definitionFile.absolutePath)
        }

orignal code:

https://github.com/oss-review-toolkit/ort/blob/e8a951080b15d38375b333d5559b651e5ef6c713/analyzer/src/main/kotlin/managers/Pip.kt#L146-L166

Sadly this did not work for me, although I do not understand why.

Could you point me into the right direction? I really need a workaround for this, until ort has a proper CLI param for it.

Additional Information:

This is being worked on already. See also: https://github.com/oss-review-toolkit/ort/issues/3671 https://github.com/oss-review-toolkit/ort/issues/3671#issuecomment-1203248523 https://github.com/oss-review-toolkit/ort/issues/4637

I noticed that the artifact files for windows, macOS and linux have quite different name formatting for the pillow package: https://pypi.org/project/Pillow/9.2.0/#files

Maybe this contributes to the issue.

sschuberth commented 2 years ago

We have some windows python tools, which are scanned with ORT for licensing purposes. Even though the tool is created for windows and scanned on linux, ORT collects the macOS artifacts of the dependencies:

This is also semi-related to the general issue described at https://github.com/oss-review-toolkit/ort/issues/4013.

pepper-jk commented 2 years ago

This is also semi-related to the general issue described at #4013.

In the merge attached to this issue I found this:

  binary_artifact:
    url: "https://files.pythonhosted.org/packages/c4/af/93fb95dc8bc90a5580084f3dc4be049d243c6f687ff2e253ed3a6df30db4/itsdangerous-2.0.0-py3-none-any.whl"

https://github.com/oss-review-toolkit/ort/pull/4011/files#diff-fdaab681c9cf677a77196de07d376e9e12656aa18af3d7d293a0d4e24f6e0088R181-R182

Can I specify this within a package configuration?

If so, this would be a possible workaround until specifying the platform is possible. It also would not require to build a modified ort-image.

sschuberth commented 2 years ago

Can I specify this within a package configuration?

You can write a (package) curation that overrides the binary artifact, yes.

mnonnenmacher commented 2 years ago

@pepper-jk We are currently waiting for https://github.com/nexB/python-inspector to provide metadata for packages. Once this is implemented, we can get independent of the local operating system and the installed Python version, and will make these available as package manager options which can be set in the .ort.yml and on the command line. This will hopefully happen soon, until then we have to rely on a combination of running local commands like pip install and pip show and running python-inspector which will not always created consistent results.

Nevertheless, I'm surprised that you get the binary artifact for Mac on a Linux machine, because IIRC we get that data from pip show which reads data from locally installed packages. If you can provide a small example project that reproduces this issue we could check if this can be fixed independently of a python-inspector upgrade.

pepper-jk commented 2 years ago

Nevertheless, I'm surprised that you get the binary artifact for Mac on a Linux machine, because IIRC we get that data from pip show which reads data from locally installed packages. If you can provide a small example project that reproduces this issue we could check if this can be fixed independently of a python-inspector upgrade.

I can confirm that I get the same macOS artifact for pillow with this example repo, that only contains our requirements.txt.

I used our production pipeline for the scan, though. It is possible that our gitlab pipeline introduces something that triggers it or the ort-config we use. I doubt the latter though, since our base config does not contain any package configuration.

@jens-erdmann maybe you can provide some information on our gitlab pipeline? I only know that they are linux kubernetes runners.

Maybe you could confirm if this issue also applies for a the plain ort docker image, @mnonnenmacher ?

pepper-jk commented 2 years ago

Hi @mnonnenmacher, today we have reproduced the problem on an Ubuntu VM outside our pipeline. Again using the example repo linked above. However, we used a more recent image 5c6fc850b0b1b0a2734e8de14c9d2b1a7b85cd73.

We ran the docker image as follows:

$ docker run --rm -it -v $SSH_AUTH_SOCK:/ssh-agent -e SSH_AUTH_SOCK=/ssh-agent -v $PWD:/project --entrypoint=/bin/bash ort-image:5c6fc850b0b1b0a2734e8de14c9d2b1a7b85cd73
$ cp -r /project ./
$ ort --info analyze -i /home/ort/project -o /home/ort/project/ort/analyzer/

And got the following metadata for pillow:9.2.0 in the analyzer-results.yml:

    - metadata:
        id: "PyPI::pillow:9.2.0"
        purl: "pkg:pypi/pillow@9.2.0"
        authors:
        - "Alex Clark (PIL Fork Author)"
        declared_licenses:
        - "HPND"
        - "Historical Permission Notice and Disclaimer (HPND)"
        declared_licenses_processed:
          spdx_expression: "HPND"
          mapped:
            Historical Permission Notice and Disclaimer (HPND): "HPND"
        description: "Python Imaging Library (Fork)"
        homepage_url: "https://python-pillow.org"
        binary_artifact:
          url: "https://files.pythonhosted.org/packages/d8/60/b13c00d403f34110e96c1b5c0afa73ce461efe3fe960c3a7e3e7fe190d82/Pillow-9.2.0-cp310-cp310-macosx_10_10_x86_64.whl"
          hash:
            value: "deb973d5f3b7dd5827667a170526d8ce"
            algorithm: "MD5"
        source_artifact:
          url: "https://files.pythonhosted.org/packages/8c/92/2975b464d9926dc667020ed1abfa6276e68c3571dcb77e43347e15ee9eed/Pillow-9.2.0.tar.gz"
          hash:
            value: "218bdb951f3e59e8b782e329ece3416d"
            algorithm: "MD5"
        vcs:
          type: ""
          url: ""
          revision: ""
          path: ""
        vcs_processed:
          type: "Git"
          url: "https://github.com/python-pillow/Pillow.git"
          revision: ""
          path: ""
      curations: []

We got the same for markupsafe:2.1.1.

The complete yml ``` --- repository: vcs: type: "Git" url: "https://github.com/pepper-jk/ort_pypi_resolution_test" revision: "3f0cafa8f4054bba83c5aedd7740b8a802bc794c" path: "" vcs_processed: type: "Git" url: "https://github.com/pepper-jk/ort_pypi_resolution_test.git" revision: "3f0cafa8f4054bba83c5aedd7740b8a802bc794c" path: "" config: {} analyzer: start_time: "2022-10-11T12:51:22.242810Z" end_time: "2022-10-11T12:51:45.725180Z" environment: ort_version: "DOCKER-SNAPSHOT-5c6fc850b0b1b0a2734e8de14c9d2b1a7b85cd73" java_version: "11.0.16.1" os: "Linux" processors: 4 max_memory: 4196401152 variables: TERM: "xterm" http_proxy: "http://proxy.com:3128" https_proxy: "http://proxy.com:3128" JAVA_HOME: "/opt/java/openjdk" ANDROID_HOME: "/opt/android-sdk" tool_versions: PIP: "22.2.2" config: allow_dynamic_versions: false result: projects: - id: "PIP::requirements.txt:3f0cafa8f4054bba83c5aedd7740b8a802bc794c" definition_file_path: "requirements.txt" declared_licenses: [] declared_licenses_processed: {} vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pepper-jk/ort_pypi_resolution_test.git" revision: "3f0cafa8f4054bba83c5aedd7740b8a802bc794c" path: "" homepage_url: "" scope_names: - "install" packages: - metadata: id: "PyPI::jinja2:3.1.2" purl: "pkg:pypi/jinja2@3.1.2" authors: - "Armin Ronacher" declared_licenses: - "BSD License" - "BSD-3-Clause" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD License: "BSD-3-Clause" description: "A very fast and expressive template engine." homepage_url: "https://palletsprojects.com/p/jinja/" binary_artifact: url: "https://files.pythonhosted.org/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl" hash: value: "00ccdb509d3592cc2163b286177d75c8" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/7a/ff/75c28576a1d900e87eb6335b063fab47a8ef3c8b4d88524c4bf78f670cce/Jinja2-3.1.2.tar.gz" hash: value: "d31148abd89c1df1cdb077a55db27d02" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pallets/jinja.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::markupsafe:2.1.1" purl: "pkg:pypi/markupsafe@2.1.1" authors: - "Armin Ronacher" declared_licenses: - "BSD License" - "BSD-3-Clause" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD License: "BSD-3-Clause" description: "Safely add untrusted strings to HTML/XML markup." homepage_url: "https://palletsprojects.com/p/markupsafe/" binary_artifact: url: "https://files.pythonhosted.org/packages/d9/60/94e9de017674f88a514804e2924bdede9a642aba179d2045214719d6ec76/MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_universal2.whl" hash: value: "edd92cc0efdc919430f86fac719864d7" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/1d/97/2288fe498044284f39ab8950703e88abbac2abbdf65524d576157af70556/MarkupSafe-2.1.1.tar.gz" hash: value: "9809f9fdd98bc835b0c21aa8f79cbf30" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pallets/markupsafe.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::pillow:9.2.0" purl: "pkg:pypi/pillow@9.2.0" authors: - "Alex Clark (PIL Fork Author)" declared_licenses: - "HPND" - "Historical Permission Notice and Disclaimer (HPND)" declared_licenses_processed: spdx_expression: "HPND" mapped: Historical Permission Notice and Disclaimer (HPND): "HPND" description: "Python Imaging Library (Fork)" homepage_url: "https://python-pillow.org" binary_artifact: url: "https://files.pythonhosted.org/packages/d8/60/b13c00d403f34110e96c1b5c0afa73ce461efe3fe960c3a7e3e7fe190d82/Pillow-9.2.0-cp310-cp310-macosx_10_10_x86_64.whl" hash: value: "deb973d5f3b7dd5827667a170526d8ce" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/8c/92/2975b464d9926dc667020ed1abfa6276e68c3571dcb77e43347e15ee9eed/Pillow-9.2.0.tar.gz" hash: value: "218bdb951f3e59e8b782e329ece3416d" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/python-pillow/Pillow.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::pyglet:1.5.26" purl: "pkg:pypi/pyglet@1.5.26" authors: - "Alex Holkner" declared_licenses: - "BSD" - "BSD License" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD: "BSD-3-Clause" BSD License: "BSD-3-Clause" description: "Cross-platform windowing and multimedia library" homepage_url: "http://pyglet.readthedocs.org/en/latest/" binary_artifact: url: "https://files.pythonhosted.org/packages/68/52/10c1826df26e59d989b115bd5ee19535cae8cbfff642f1495fc62f24b01f/pyglet-1.5.26-py3-none-any.whl" hash: value: "f1da62163a65ee518469c210058dd3c9" algorithm: "MD5" source_artifact: url: "https://files.pythonhosted.org/packages/85/5d/2d52c9441344802f081d1cda30839e5534845d52048629188bafba475444/pyglet-1.5.26.zip" hash: value: "61da58d945346cc9adbd3acd1e4fd8dd" algorithm: "MD5" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pyglet/pyglet.git" revision: "" path: "" curations: [] - metadata: id: "PyPI::pywavefront:1.3.3" purl: "pkg:pypi/pywavefront@1.3.3" authors: - "Kurt Yoder" declared_licenses: - "BSD" - "BSD License" declared_licenses_processed: spdx_expression: "BSD-3-Clause" mapped: BSD: "BSD-3-Clause" BSD License: "BSD-3-Clause" description: "Python library for importing Wavefront .obj files" homepage_url: "https://github.com/pywavefront/PyWavefront" binary_artifact: url: "https://files.pythonhosted.org/packages/fe/f2/db25754f3d7f948e1cb7191a393284e29bc2497dae9d8456cbf25ddac671/PyWavefront-1.3.3-py3-none-any.whl" hash: value: "fb87b53d8e143a7d0f057c1ca4575f4f" algorithm: "MD5" source_artifact: url: "" hash: value: "" algorithm: "" vcs: type: "" url: "" revision: "" path: "" vcs_processed: type: "Git" url: "https://github.com/pywavefront/PyWavefront.git" revision: "" path: "" curations: [] dependency_graphs: PIP: packages: - "PyPI::jinja2:3.1.2" - "PyPI::markupsafe:2.1.1" - "PyPI::pillow:9.2.0" - "PyPI::pyglet:1.5.26" - "PyPI::pywavefront:1.3.3" scopes: :requirements.txt:3f0cafa8f4054bba83c5aedd7740b8a802bc794c:install: - root: 0 - root: 2 - root: 3 - root: 4 nodes: - pkg: 1 - {} - pkg: 2 - pkg: 3 - pkg: 4 edges: - from: 1 to: 0 has_issues: false scanner: null advisor: null evaluator: null ```
sschuberth commented 2 years ago

However, we used a more recent image 5c6fc85.

@pepper-jk this should have been improved just today with https://github.com/oss-review-toolkit/ort/pull/5910. Please give this another try.

pepper-jk commented 2 years ago

Thanks, we will. The image build of the current master is running.

I will update you tomorrow.

pepper-jk commented 2 years ago

I can now confirm that a scan with ORT on ec7d0b1 does not have the macOS binary artifact problem anymore.

I assume #5910 fixed this bug.

We now get the following metadata for pillow:9.2.0 in the analyzer-results.yml:

    - metadata:
        id: "PyPI::pillow:9.2.0"
        purl: "pkg:pypi/pillow@9.2.0"
        authors:
        - "Alex Clark (PIL Fork Author) <aclark@python-pillow.org>"
        declared_licenses:
        - "HPND"
        declared_licenses_processed:
          spdx_expression: "HPND"
        description: "<p align=\"center\">"
        homepage_url: "https://python-pillow.org"
        binary_artifact:
          url: "https://files.pythonhosted.org/packages/c4/ac/a50a4a11fe2120d3047b567f765afb54d6c57bad704e8c9759153b6359e4/Pillow-9.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
          hash:
            value: "1cc1d2451e8a3b4bfdb9caf745b58e6c7a77d2e469159b0d527a4554d73694d1"
            algorithm: "SHA-256"
        source_artifact:
          url: "https://files.pythonhosted.org/packages/8c/92/2975b464d9926dc667020ed1abfa6276e68c3571dcb77e43347e15ee9eed/Pillow-9.2.0.tar.gz"
          hash:
            value: "75e636fd3e0fb872693f23ccb8a5ff2cd578801251f3a4f6854c6a5d437d3c04"
            algorithm: "SHA-256"
        vcs:
          type: ""
          url: ""
          revision: ""
          path: ""
        vcs_processed:
          type: "Git"
          url: "https://github.com/python-pillow/Pillow.git"
          revision: ""
          path: ""
      curations: []

Same goes for markupsafe:2.1.1.

pepper-jk commented 2 years ago

I will keep the issue open though, until passing a platform manually is possible in ORT.

mnonnenmacher commented 2 years ago

@pepper-jk This was already implemented in #5910. You can configure the Python version or operating system either in the global configuration or in your .ort.yml like this:

analyzer:
  packageManagers: # package_managers in .ort.yml IIRC
    Pip:
      options:
        operatingSystem: linux
        pythonVersion: 3.8
pepper-jk commented 2 years ago

Cool, I will test it next week.

pepper-jk commented 2 years ago

I added this .ort.yml to the test repo:

analyzer:
  packageManagers: # package_managers in .ort.yml IIRC
    Pip:
      options:
        operatingSystem: windows
        pythonVersion: 3.10

https://github.com/pepper-jk/ort_pypi_resolution_test/blob/main/.ort.yml

Sadly running a scan with ORT using this config and ORT on 7d6146ef419db9b658e373a1e3055e7f9fce519b did not work:

ort@d287c2beb547:~$ /config/scan.sh
09:40:58,672 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.4
09:40:58,703 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
09:40:58,714 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml]
09:40:58,715 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs multiple times on the classpath.
09:40:58,716 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml]
09:40:58,716 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/cli-7d6146ef41-dirty.jar!/logback.xml]
09:40:58,725 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@69da0b12 - URL [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml] is not of type file
09:40:58,841 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
09:40:58,841 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
09:40:58,854 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
09:40:58,876 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOTlogger to WARN
09:40:58,880 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.headers] to ERROR
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.wire] to ERROR
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.eclipse.jgit.internal.storage.file.FileSnapshot] to ERROR
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.analyzer.managers.Yarn2] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.clients.fossid.FossIdRestService] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.reporter.reporters.fossid.FossIdReporter] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossId] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossIdConfig] to INFO
09:40:58,881 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossIdUrlProvider] to INFO
09:40:58,881 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@2764c546 - End of configuration.
09:40:58,882 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@59496961 - Registering current configuration as safe fallback point

________ _____________________
\_____  \\______   \__    ___/ the OSS Review Toolkit, version 7d6146ef41-dirty.
 /   |   \|       _/ |    |
/    |    \    |   \ |    |    Running 'analyze' under Java 11.0.16.1 on Linux with
\_______  /____|_  / |____|    4 CPUs and a maximum of 4002 MiB of memory.
        \/       \/
Environment variables:
ORT_CONFIG_DIR = /home/ort/.ort/config
ORT_DATA_DIR = /home/ort/.ort
TERM = xterm
http_proxy = http://proxy.com:3128
https_proxy = http://proxy.com:3128
JAVA_HOME = /opt/java/openjdk
ANDROID_HOME = /opt/android-sdk

Looking for analyzer-specific configuration in the following files and directories:
        /home/ort/.ort/config/curations.yml (does not exist)
        /home/ort/.ort/config/curations (does not exist)
        /home/ort/project/.ort.yml
        /home/ort/.ort/config/resolutions.yml (does not exist)
The following package managers are enabled:
        Bower, Bundler, Cargo, Carthage, CocoaPods, Composer, Conan, DotNet, GoDep, GoMod, Gradle, Maven, NPM, NuGet, PIP, PNPM, Pipenv, Poetry, Pub, SBT, SpdxDocumentFile, Stack, Unmanaged, Yarn, Yarn2
Analyzing project path:
        /home/ort/project
Exception in thread "main" java.lang.IllegalArgumentException: Unrecognized field "packageManagers" (class org.ossreviewtoolkit.model.config.AnalyzerConfiguration), not marked as ignorable (5 known properties: "sw360_configuration", "allow_dynamic_versions", "disabled_package_managers", "enabled_package_managers", "package_managers"])
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: org.ossreviewtoolkit.model.config.RepositoryConfiguration["analyzer"]->org.ossreviewtoolkit.model.config.AnalyzerConfiguration["packageManagers"])
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4393)
        at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4334)
        at org.ossreviewtoolkit.cli.commands.AnalyzerCommand.run(AnalyzerCommand.kt:591)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
        at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:82)
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "packageManagers" (class org.ossreviewtoolkit.model.config.AnalyzerConfiguration), not marked as ignorable (5 known properties: "sw360_configuration", "allow_dynamic_versions", "disabled_package_managers", "enabled_package_managers", "package_managers"])
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: org.ossreviewtoolkit.model.config.RepositoryConfiguration["analyzer"]->org.ossreviewtoolkit.model.config.AnalyzerConfiguration["packageManagers"])
        at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1127)
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2023)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1700)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperties(BeanDeserializerBase.java:1650)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:540)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:564)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4388)
        ... 10 more
09:41:01,059 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.4
09:41:01,073 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
09:41:01,079 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml]
09:41:01,080 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs multiple times on the classpath.
09:41:01,080 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml]
09:41:01,080 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@73ad4ecc - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/cli-7d6146ef41-dirty.jar!/logback.xml]
09:41:01,086 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@69da0b12 - URL [jar:file:/opt/ort/lib/helper-cli-7d6146ef41-dirty.jar!/logback.xml] is not of type file
09:41:01,158 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
09:41:01,158 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
09:41:01,171 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
09:41:01,190 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to WARN
09:41:01,194 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
09:41:01,195 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.headers] to ERROR
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.wire] to ERROR
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.eclipse.jgit.internal.storage.file.FileSnapshot] to ERROR
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.analyzer.managers.Yarn2] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.clients.fossid.FossIdRestService] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.reporter.reporters.fossid.FossIdReporter] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossId] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossIdConfig] to INFO
09:41:01,196 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.scanner.scanners.fossid.FossIdUrlProvider] to INFO
09:41:01,196 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@2764c546 - End of configuration.
09:41:01,196 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@59496961 - Registering current configuration as safe fallback point
mnonnenmacher commented 2 years ago

@pepper-jk In the .ort.yml please use package_managers instead of packageManagers. Unfortunately we use different parsers for the main configuration file and the .ort.yml and we still have to align them in this regard.

pepper-jk commented 2 years ago

Oh, sorry I did not get that comment. :see_no_evil:

When I try it with package_managers, it complains about the python version:

12:51:53.728 [main] INFO  org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/home/ort/pr
oject/.git' as it is hard-coded to be ignored.
Exception in thread "main" java.lang.IllegalArgumentException: The 'pythonVersion' option must be one of '2.7'
, '3.6', '3.7', '3.8', '3.9', '3.10'.
        at org.ossreviewtoolkit.analyzer.managers.Pip.<init>(Pip.kt:93)
        at org.ossreviewtoolkit.analyzer.managers.Pip$Factory.create(Pip.kt:81)
        at org.ossreviewtoolkit.analyzer.managers.Pip$Factory.create(Pip.kt:74)
        at org.ossreviewtoolkit.analyzer.Analyzer.findManagedFiles(Analyzer.kt:89)
        at org.ossreviewtoolkit.cli.commands.AnalyzerCommand.run(AnalyzerCommand.kt:263)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
        at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:82)

I tried 3.10, 3.9, and 3.8 with windows as the platform, and 3.8 with linux. Neither worked for me.

Is it possible the version input verification is buggy?

sschuberth commented 2 years ago

Is it possible the version input verification is buggy?

Can you try quoting the version? I.e. use pythonVersion: "3.10" to avoid the value being parsed as a floating point number.

sschuberth commented 2 years ago

Is it possible the version input verification is buggy?

Can you try quoting the version? I.e. use pythonVersion: "3.10" to avoid the value being parsed as a floating point number.

Scratch that. Instead, "PIP" instead of "Pip" in the config. I'm going to relax that.

sschuberth commented 2 years ago

Is it possible the version input verification is buggy?

Can you try quoting the version? I.e. use pythonVersion: "3.10" to avoid the value being parsed as a floating point number.

Scratch that. Instead, "PIP" instead of "Pip" in the config. I'm going to relax that.

LOL, that's also not it, as the lookup already is case-insensitive. I'll try to improve the error message to see what's going on.

sschuberth commented 2 years ago

I'll try to improve the error message to see what's going on.

Please try with https://github.com/oss-review-toolkit/ort/pull/5956. Maybe there's a trailing space or something like that?

pepper-jk commented 2 years ago

Ah, I forgot to mention that I tried the version and platform in quotes ' but I believe only 'windows' and '3.10'.

I doubt that there are trailing whitespaces, as my vscode is configured to delete those on save, but I will check it tomorrow.

I will also test it with your fix tomorrow.

Thanks for the help.

pepper-jk commented 2 years ago

I have run a scan with the new main branch. I now see what the issue is. It reports the version 3.1 apparently truncating the 0 in 3.10:

Exception in thread "main" java.lang.IllegalArgumentException: The 'pythonVersion' option must be one of '2.7', '3.6', '3.7', '3.8', '3.9', '3.10', but was '3.1'.
        at org.ossreviewtoolkit.analyzer.managers.Pip.<init>(Pip.kt:94)
        at org.ossreviewtoolkit.analyzer.managers.Pip$Factory.create(Pip.kt:81)
        at org.ossreviewtoolkit.analyzer.managers.Pip$Factory.create(Pip.kt:74)
        at org.ossreviewtoolkit.analyzer.Analyzer.findManagedFiles(Analyzer.kt:89)
        at org.ossreviewtoolkit.cli.commands.AnalyzerCommand.run(AnalyzerCommand.kt:263)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
        at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:82)

I also ran the scan with 3.9, which works fine now. I am not sure, if you fixed something, or if I just forgot to copy the config file into the directory I was scanning (after chaging the version(s) the last time).

sschuberth commented 2 years ago

It reports the version 3.1 apparently truncating the 0 in 3.10:

But that should not happen when quoting the value as indicated above, or?

pepper-jk commented 2 years ago

Yes, this test was without quotation.

If I put the version in quotes '3.10' it works. Thanks.

Closing this issue.