Provide more specific stack trace for config.yml parse error?

[schvvarzekatze]

Hello, I have problems on parsing my project specific config.yml . It seems to caused by an indentation error.

As it is hard to find the belonging line I would like to ask if it is possible also to provide the specific error category or at least the line in config.yml as it is the case when parsing NOTICE file templates.

Thank you very much.

13:43:43.361 [main] INFO  org.ossreviewtoolkit.model.config.OrtConfiguration - Using ORT configuration file '/project/ort/config.yml'.
Exception in thread "main" java.lang.IllegalArgumentException: Failed to load ORT configuration: **Could not parse /project/ort/config.yml**
    at org.ossreviewtoolkit.model.config.OrtConfiguration$Companion$load$1.invoke(OrtConfiguration.kt:156)
    at org.ossreviewtoolkit.model.config.OrtConfiguration$Companion$load$1.invoke(OrtConfiguration.kt:155)
    at com.sksamuel.hoplite.fp.ValidatedKt.getOrElse(Validated.kt:115)
    at org.ossreviewtoolkit.model.config.OrtConfiguration$Companion.load(OrtConfiguration.kt:155)
    at org.ossreviewtoolkit.cli.OrtMain.run(OrtMain.kt:171)
    at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
    at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
    at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
    at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
    at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
    at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
    at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:76)

[sschuberth]

I guess we're more or less stuck with what Hoplite tells us, and that seems to be no more than "Could not parse /project/ort/config.yml". If it fails that hard, the file seems to be completely wrong. Is it maybe still in HOCON format instead of YAML?

[schvvarzekatze]

It is shown as .yml in IntelliJ like curations.yml

[sschuberth]

That does not really mean anything... it could be a HOCON (*.conf) file that has been renamed to *.yml.

[schvvarzekatze]

I think you are right because actually I did this.

[schvvarzekatze]

The stack trace also comes with only one indentation error. So the only way to find the incorrect indentation seems to install a plugin e.g. Yamllint for IntelliJ which highlights the wrong line.

[sschuberth]

I think you are right because actually I did this.

So, if you just renamed ort.conf to config.yml this cannot work. You also need to convert the file's contents from HOCON to YAML. The ort config --hocon-to-yaml command can assist with this.

[sschuberth]

@schvvarzekatze is your issue solved? If so, can you share your config.yml (with private data replaced)?

[schvvarzekatze]

If it is not possible to print out the line of the parsing error in the yaml my issue is solved.

# Copyright (C) 2022 The ORT Project Authors (see <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
# License-Filename: LICENSE

# A reference configuration file containing all possible ORT configuration options. Some of those options are mutually
# exclusive, so this file is only used to show all options and to validate the configuration.

ort:
  allowedProcessEnvironmentVariableNames:
    - PASSPORT
    - USER_HOME
  deniedProcessEnvironmentVariablesSubstrings:
    - PASS
    - SECRET
    - TOKEN
    - USER
  enableRepositoryPackageCurations: true
  enableRepositoryPackageConfigurations: true

  licenseFilePatterns:
    licenseFilenames: [ 'license*' ]
    patentFilenames: [ patents ]
    rootLicenseFilenames: [ 'readme*' ]

  severeIssueThreshold: ERROR
  severeRuleViolationThreshold: ERROR

  analyzer:
    allowDynamicVersions: true
    enabledPackageManagers: [ Gradle, Yarn ]
    packageManagers:
      Yarn:
        options:
          directDependenciesOnly: true
      Gradle:
        options:
          directDependenciesOnly: true

  downloader:
    allowMovingRevisions: true
    # Only used if the '--license-classifications-file' option is specified.
    includedLicenseCategories:
      - copyleft
      - copyleft-provide-sourcecode
      - weak-copyleft
      - weak-copyleft-provide-sourcecode
      - proprietary
      - permissive
      - public-domain
      - no-assertion
      - not-for-commercial-use
      - include-in-notice-file
      - include-source-code-offer-in-notice-file
      -
  scanner:
    skipConcluded: true

    archive:
      enabled: false

      postgresStorage:
        connection:
          url: ${POSTGRES_URL}
          schema: public
          username: ort
          password: ${POSTGRES_PASSWORD}

    createMissingArchives: false

    # Map scanner license findings to valid SPDX licenses. Note that these mappings are only applied in new scans,
    # stored scan results are not affected.
    detectedLicenseMapping:
      LGPL v2.1: 'LGPL-2.1-only'

    options:
      ScanCode:
        commandLine: '--copyright --license --info --strip-root --timeout 300'
        parseLicenseExpressions: true
        # Criteria for matching stored scan results. These can be configured for any scanner that uses semantic
        # versioning. Note that the 'maxVersion' is exclusive and not part of the range of accepted versions.
        minVersion: '3.2.1-rc2'
        maxVersion: '32.0.0'

    storages:
      postgres:
        connection:
          url: ${POSTGRES_URL}
          schema: public
          username: ort
          password: ${POSTGRES_PASSWORD}
        type: PACKAGE_BASED

    storageReaders: [ postgres ]
    storageWriters: [ postgres ]

    ignorePatterns: [ '**/META-INF/DEPENDENCIES' ]

    provenanceStorage:
      postgresStorage:
        connection:
          url: ${POSTGRES_URL}
          schema: public
          username: ort
          password: ${POSTGRES_PASSWORD}

[sschuberth]

Saving your config as config.yml and running ort config --check-syntax config.yml says the syntax is valid. So I guess that in your environment one of the variables you're referring to is unset / set to an empty string, which could render the file invalid.

[sschuberth]

Or maybe, if you password starts with a special char like "*", that could also lead to problems as the variable is unquoted.

[schvvarzekatze]

Yes, this config.yml has been valid. But if I change only one indentation there should be a parsing error.

[sschuberth]

But if I change only one indentation there should be a parsing error.

Right. But in that case you should get a pretty sophisticated error message (incl. line number) like

    - 'ort': - Could not instantiate 'org.ossreviewtoolkit.model.config.OrtConfiguration' because:
        - 'scanner': - Could not instantiate 'org.ossreviewtoolkit.model.config.ScannerConfiguration' because:
            - 'storages': Collection element decode failure (config.yml:97:6):
                Could not find appropriate subclass of class org.ossreviewtoolkit.model.config.ScanStorageConfiguration: Tried org.ossreviewtoolkit.model.config.ClearlyDefinedStorageConfiguration, org.ossreviewtoolkit.model.config.FileBasedStorageConfiguration, org.ossreviewtoolkit.model.config.PostgresStorageConfiguration, org.ossreviewtoolkit.model.config.Sw360StorageConfiguration (config.yml:103:12)

in most cases.

[schvvarzekatze]

I never got the stack trace like this. Did you get it with: ort config --check-syntax config.yml?

[sschuberth]

Did you get it with: ort config --check-syntax config.yml?

Yes, but you would also get it with regular ORT subcommands. For testing, I de-indented type: PACKAGE_BASED in line 104 of your example by 2 spaces.

[schvvarzekatze]

In my case I do not get this sophisticated error message with regular ORT subcommands, but I did not define any Sw360StorageConfiguration.

[sschuberth]

FYI, this upgrade of Hoplite brings more detailed YAML error messages.