search

oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.61k stars 312 forks source link

config.yml is getting ignored while physically available #9490

Open MNesche opened 4 days ago

MNesche commented 4 days ago

Describe the bug

Running ORT (any Version, starting from 28.0) in Jenkins on Windows, the config.yml (format previously to ORT Version 41.0.0) is physically present but will not be parsed further to be used for the execution. The default configuration settings are used instead. When implementing a false intendation in the config.yml, the ORT commands fail, due to the wrong intendation. This shows that the config.yml can be accessed and is also read by ORT.

Since ORT Version 41.0.0, the key "rootLicenseFilenames" in "licenseFilePatterns" of the config.yml got renamed. That should throw an error, cause the key is not valid anymore, if the same config is used with ORT 41.0.0. However, there is no error according the invalid key when running any ORT command, the fallback standard configuration settings are used instead.

To Reproduce

Steps to reproduce the behavior:

  1. Install ORT on Jenkins with the right Environment Variables
  2. Change specific values in the config.yml, i.e. limit the enabledPackageManagers
  3. Execute a command like "ort config --show-active"
  4. The content of the values are the standard values; in the example above, all possible PackageManagers are enabled.

Expected behavior

Settings in the config.yml will be applied successfully.

Console / log output

Add console and / or log output that shows the error and additional context. No screenshots of plain text please, to keep text searchable.

Output after using command "ort config --show-active":

c:\FileDir>"c:\jenkins-slave\tools\com.cloudbees.jenkins.plugins.customtools.CustomTool\ORT41\bin\ort.bat" --info config --show-active  
17:07:43.709 [main] INFO  org.ossreviewtoolkit.model.config.OrtConfiguration - Using ORT configuration file 'c:\FileDir\.ort\ort-config\config.yml'.
Hoplite is configured to infer which sealed type to choose by inspecting the config values at runtime. This behaviour is now deprecated in favour of explicitly specifying the type through a discriminator field. In 3.0 this new behavior will become the default. To enable this behavior now (and disable this warning), invoke withExplicitSealedTypes() on the ConfigLoaderBuilder.
17:07:44.455 [main] INFO  org.ossreviewtoolkit.utils.common.EnvironmentVariableFilter - EnvironmentVariableFilter initialized with denySubstrings = [key, pass, pwd, token, user] and allowNames = [CARGO_HTTP_USER_AGENT, COMPOSER_ALLOW_SUPERUSER, CONAN_LOGIN_ENCRYPTION_KEY, CONAN_LOGIN_USERNAME, CONAN_PASSWORD, CONAN_USERNAME, CONAN_USER_HOME, CONAN_USER_HOME_SHORT, DOTNET_CLI_CONTEXT_ANSI_PASS_THRU, GIT_ASKPASS, GIT_HTTP_USER_AGENT, GRADLE_USER_HOME, HACKAGE_USERNAME, HACKAGE_PASSWORD, HACKAGE_KEY, PWD, USER, USERPROFILE].
 ______________________________                                                
/        \_______   \__    ___/        The OSS Review Toolkit, version 41.0.0, 
|    |   | |       _/ |    |           built with JDK 21.0.5+11-LTS, running un
|    |   | |    |   \ |    |           Executing 'config' as 'ort-user' on 
\________/ |____|___/ |____|           with 4 CPUs and a maximum of 3066 MiB of

Environment variables:                                                        
ORT_CONFIG_DIR = c:\FileDir\.ort\ort-config  
ORT_DATA_DIR = c:\FileDir\.ort               
USERPROFILE = C:\Users\ort-user                                           
OS = Windows_NT                                                               
COMSPEC = C:\WINDOWS\system32\cmd.exe                                         
JAVA_HOME = c:\jenkins-slave\tools\hudson.model.JDK\java21_x86_64             

Looking for ORT configuration in the following file:
        c:\FileDir\.ort\ort-config\config.yml

The active configuration is:

ort:
  addAuthorsToCopyrights: false
  allowedProcessEnvironmentVariableNames:
  - "CARGO_HTTP_USER_AGENT"
  - "COMPOSER_ALLOW_SUPERUSER"
  - "CONAN_LOGIN_ENCRYPTION_KEY"
  - "CONAN_LOGIN_USERNAME"
  - "CONAN_PASSWORD"
  - "CONAN_USERNAME"
  - "CONAN_USER_HOME"
  - "CONAN_USER_HOME_SHORT"
  - "DOTNET_CLI_CONTEXT_ANSI_PASS_THRU"
  - "GIT_ASKPASS"
  - "GIT_HTTP_USER_AGENT"
  - "GRADLE_USER_HOME"
  - "HACKAGE_USERNAME"
  - "HACKAGE_PASSWORD"
  - "HACKAGE_KEY"
  - "PWD"
  - "USER"
  - "USERPROFILE"
  deniedProcessEnvironmentVariablesSubstrings:
  - "key"
  - "pass"
  - "pwd"
  - "token"
  - "user"
  enableRepositoryPackageConfigurations: false
  enableRepositoryPackageCurations: false
  forceOverwrite: false
  licenseFilePatterns:
    licenseFilenames:
    - "copying*"
    - "copyright"
    - "licence*"
    - "license*"
    - "*.licence"
    - "*.license"
    - "unlicence"
    - "unlicense"
    patentFilenames:
    - "patents"
    otherLicenseFilenames:
    - "readme*"
  packageConfigurationProviders:
  - type: "DefaultDir"
    id: "DefaultDir"
    enabled: true
    options: {}
  packageCurationProviders:
  - type: "DefaultDir"
    id: "DefaultDir"
    enabled: true
    options: {}
  - type: "DefaultFile"
    id: "DefaultFile"
    enabled: true
    options: {}
  severeIssueThreshold: "WARNING"
  severeRuleViolationThreshold: "WARNING"
  analyzer:
    allowDynamicVersions: false
    skipExcluded: false
  advisor:
    skipExcluded: false
  downloader:
    allowMovingRevisions: false
    includedLicenseCategories: []
    skipExcluded: false
    sourceCodeOrigins:
    - "VCS"
    - "ARTIFACT"
  scanner:
    skipConcluded: false
    skipExcluded: false
    detectedLicenseMapping:
      LicenseRef-scancode-agpl-generic-additional-terms: "NOASSERTION"
      LicenseRef-scancode-free-unknown: "NOASSERTION"
      LicenseRef-scancode-generic-cla: "NOASSERTION"
      LicenseRef-scancode-generic-exception: "NOASSERTION"
      LicenseRef-scancode-generic-export-compliance: "NOASSERTION"
      LicenseRef-scancode-generic-tos: "NOASSERTION"
      LicenseRef-scancode-generic-trademark: "NOASSERTION"
      LicenseRef-scancode-gpl-generic-additional-terms: "NOASSERTION"
      LicenseRef-scancode-other-copyleft: "NOASSERTION"
      LicenseRef-scancode-other-permissive: "NOASSERTION"
      LicenseRef-scancode-patent-disclaimer: "NOASSERTION"
      LicenseRef-scancode-unknown: "NOASSERTION"
      LicenseRef-scancode-unknown-license-reference: "NOASSERTION"
      LicenseRef-scancode-unknown-spdx: "NOASSERTION"
      LicenseRef-scancode-warranty-disclaimer: "NOASSERTION"
    ignorePatterns:
    - "**/*.ort.yml"
    - "**/*.spdx.yml"
    - "**/*.spdx.yaml"
    - "**/*.spdx.json"
    - "**/META-INF/DEPENDENCIES"
    - "**/META-INF/DEPENDENCIES.txt"
    - "**/META-INF/NOTICE"
    - "**/META-INF/NOTICE.txt"
  reporter: {}
  notifier:
    mail: null
    jira: null

Environment

Output of the ort requirements -l commands command:

 ______________________________                                                
/        \_______   \__    ___/        The OSS Review Toolkit, version 41.0.0, 
|    |   | |       _/ |    |           built with JDK 21.0.5+11-LTS, running un
|    |   | |    |   \ |    |           Executing 'requirements' as 'exam-servic
\________/ |____|___/ |____|           with 4 CPUs and a maximum of 3066 MiB of

Environment variables:                                                        
ORT_CONFIG_DIR = c:\FileDir\.ort\ort-config  
ORT_DATA_DIR = c:\FileDir\.ort               
USERPROFILE = C:\Users\ort-user                                           
OS = Windows_NT                                                               
COMSPEC = C:\WINDOWS\system32\cmd.exe                                         
JAVA_HOME = c:\jenkins-slave\tools\hudson.model.JDK\java21_x86_64

Or manually specify:

And specify (relevant parts of) your ORT configuration (config.yml):

ort:
  allowedProcessEnvironmentVariableNames:
    - PASSPORT
    - USER_HOME
  deniedProcessEnvironmentVariablesSubstrings:
    - PASS
    - SECRET
    - TOKEN
    - USER

  enableRepositoryPackageConfigurations: true
  enableRepositoryPackageCurations: true

  # Force overwriting of any existing output files.
  forceOverwrite: true

  licenseFilePatterns:
    licenseFilenames: ['license*']
    patentFilenames: ['patents']
    rootLicenseFilenames: ['readme*']

Additional context

Further investigation assumes a problem in the hoplite library. The corresponding line of the issue happening in the source code of ORT seems to be at: https://github.com/oss-review-toolkit/ort/blob/1d5676fb98b833b0382c4be57d66d41355ad65e0/model/src/main/kotlin/config/OrtConfiguration.kt#L173

sschuberth commented 4 days ago

Looks like there's a subtle issue with indentation of YAML lists: The "-" for the item must be on the start column of the parent key. This works for me:

ort:
  allowedProcessEnvironmentVariableNames:
  - PASSPORT
  - USER_HOME
  deniedProcessEnvironmentVariablesSubstrings:
  - PASS
  - SECRET
  - TOKEN
  - USER

  enableRepositoryPackageConfigurations: true
  enableRepositoryPackageCurations: true

  # Force overwriting of any existing output files.
  forceOverwrite: true

  licenseFilePatterns:
    licenseFilenames: ['license*']
    patentFilenames: ['patents']
    otherLicenseFilenames: ['readme*']
sschuberth commented 4 days ago

Very weird, I was able to reproduce it once, but now not anymore...