search

oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.61k stars 309 forks source link

How to provide link to Binary and Source Artifact in customized NOTICE template #4959

Closed schvvarzekatze closed 2 years ago

schvvarzekatze commented 2 years ago

How could I provide the link to Binary and Source Artifact in NOTICE template?

In FreemarkerTemplateProcessor.kt there seems not to be a class with such functions.

sschuberth commented 2 years ago

There doesn't need to be a dedicated function for that, as the processor as access to all of there properties directly:

https://github.com/oss-review-toolkit/ort/blob/67a944262f7cec9b6e2029fc8f1d73b90ecae037/reporter/src/main/kotlin/reporters/freemarker/FreemarkerTemplateProcessor.kt#L123-L130

SO you can simply iterate over the packages and access each package's RemoteArtifacts:

https://github.com/oss-review-toolkit/ort/blob/67a944262f7cec9b6e2029fc8f1d73b90ecae037/model/src/main/kotlin/Package.kt#L96-L104

schvvarzekatze commented 2 years ago

@sschuberth : Thank you very much for your explanation. The iterating works partly and makes the .ftl compilable. But it does not fetch several values of the package, even if they exist in evaluation-result.yml which is the input for the report.

Example: Package in evaluation-result.yml

- package:
        id: "Maven:ch.qos.logback:logback-classic:1.2.5"
        purl: "pkg:maven/ch.qos.logback/logback-classic@1.2.5"
        authors:
        - "Ceki Gulcu"
        - "Joern Huxhorn"
        - "QOS.ch"
        declared_licenses:
        - "Eclipse Public License - v 1.0"
        - "GNU Lesser General Public License"
        declared_licenses_processed:
          spdx_expression: "EPL-1.0 AND LGPL-2.1-or-later"
          mapped:
            Eclipse Public License - v 1.0: "EPL-1.0"
            GNU Lesser General Public License: "LGPL-2.1-or-later"
        concluded_license: "LGPL-2.1-only"
        description: "logback-classic module"
        homepage_url: "http://logback.qos.ch/logback-classic"
        binary_artifact:
          url: "https://repo.maven.apache.org/maven2/ch/qos/logback/logback-classic/1.2.5/logback-classic-1.2.5.jar"
          hash:
            value: "030e0c3932f24fb10e7851dd308a3ad14e570d60"
            algorithm: "SHA-1"
        source_artifact:
          url: "https://repo.maven.apache.org/maven2/ch/qos/logback/logback-classic/1.2.5/logback-classic-1.2.5-sources.jar"
          hash:
            value: "320626cebb9a3f221ce82f03ffc9340bbcd48fac"
            algorithm: "SHA-1"
        vcs:
          type: "Git"
          url: "git@github.com:ceki/logback.git"
          revision: ""
          path: ""
        vcs_processed:
          type: "Git"
          url: "ssh://git@github.com/ceki/logback.git"
          revision: ""
          path: ""
      curations:
      - base: {}
        curation:
          concluded_license: "LGPL-2.1-only"
          declared_license_mapping:
            EPL-1.0 AND LGPL-2.1-only: "LGPL-2.1-only"

customized_license_report.ftl

[#list packages as package]
    [#if !package.excluded]
        ----

        Package: [#if package.id.namespace?has_content]${package.id.namespace}:[/#if]${package.id.name}:${package.id.version}

        [#if package.binaryArtifact?has_content && package.binaryArtifact.url?has_content]
            [#assign binaryArtifact = package.binaryArtifact.url]
            [#if binaryArtifact?has_content]
                Binary Artifact: ${binaryArtifact}
            [/#if]
        [/#if]
        [#if package.sourceArtifact?has_content && package.sourceArtifact.url?has_content]
            [#assign sourceArtifact = package.sourceArtifact.url]
            [#if sourceArtifact?has_content]
                Source Artifact: ${sourceArtifact}
            [/#if]
        [/#if]
         [#if package.id?has_content]
            [#assign pid = package.id]
            [#if pid?has_content]
                Package Id: ${pid}
            [/#if]
        [/#if]
    [/#if]
[/#list]

In the report it only works for id:

Package: ch.qos.logback:logback-classic:1.2.5

                Package Id: Identifier(type=Maven, namespace=ch.qos.logback, name=logback-classic, version=1.2.5)

                Package Info: org.ossreviewtoolkit.reporter.reporters.freemarker.FreemarkerTemplateProcessor$PackageModel@543ece68
schvvarzekatze commented 2 years ago

@sschuberth : Does the NOTICE File have to do something with ort.config if there are values in the attributes? (With ClearlyDefined it also doesn't retreive the package values) Or should it always retreive the values from evaluation-result.yml as the reporter uses this result?

sschuberth commented 2 years ago

I'm not sure if I'm getting your question right, but in general any reporter gets the ReporterInput as input, which includes the OrtResult (which may be an evaluator result) and the OrtConfiguration. You can freely access any information from these that you need to create your report, and that also works via FreeMarker template files.

schvvarzekatze commented 2 years ago

Thank you. This gives me at least to find out another way to retreive the Source Artifacts for Maven and Github:

[#list resolvedLicenses as resolvedLicense]
        [#assign licenseName = resolvedLicense.license.simpleLicense()]
        [#assign licenseLocations = resolvedLicense.locations]
        [#assign licenseText = licenseTextProvider.getLicenseText(licenseName)!]
        [#if !licenseText?has_content][#continue][/#if]
        [#if isFirst]
            [#if !hasNoticePackageLicenses]
                ----

                Package: [#if package.id.namespace?has_content]${package.id.namespace}:[/#if]${package.id.name}:${package.id.version}
                [#assign hasNoticePackageLicenses = true]
            [/#if]

            [#if licenseLocations?has_content && licenseLocations[0].provenance?has_content]
            [#assign artifactProvenance = licenseLocations[0].provenance]
                [#if artifactProvenance?has_content && licenseLocations[0].provenance.sourceArtifact?has_content]
                Source Artifact: ${artifactProvenance.sourceArtifact.url}
                [/#if]
                [#if artifactProvenance?has_content && licenseLocations[0].provenance.vcsInfo?has_content]
                Source Artifact: ${artifactProvenance.vcsInfo.url}
                [/#if]
            [#else]
                No Source Artifact available
            [/#if]

            The following copyrights and licenses were found in the source code of this package:
            [#assign isFirst = false]
        [#else]
            --
        [/#if]
        [#assign copyrights = resolvedLicense.getCopyrights()]
        [#if copyrights?has_content]

        [/#if]
        ${copyrights?join("\n", "", "\n")}
        ${licenseText}

        [#assign exceptionName = resolvedLicense.license.exception()!]
        [#assign exceptionText = licenseTextProvider.getLicenseText(exceptionName)!]
        [#if exceptionText?has_content]
            ${exceptionText}
        [/#if]
    [/#list]

But I have still in about 10% of the packages no Source Artifact Location, e.g. com.codepoetics:protonpack:1.16

For this package I would have expected to find the url of the Source Artifact. The evaluation-result.yml which is processed by the Reporter contains the following:

provenance:
          source_artifact:
            url: "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16-sources.jar"
            hash:
              value: "3ddc16c880eefdf48354c1208cb12cce0e8189a7"
              algorithm: "SHA-1"

Do you have an explanation?

schvvarzekatze commented 2 years ago

I also tried to reproduce it on an empty sample project only containing the protonpack dependency:

Here the steps:

Analyzer result:

{
  "repository" : {
    "vcs" : {
      "type" : "",
      "url" : "",
      "revision" : "",
      "path" : ""
    },
    "vcs_processed" : {
      "type" : "",
      "url" : "",
      "revision" : "",
      "path" : ""
    },
    "config" : { }
  },
  "analyzer" : {
    "start_time" : "2022-09-02T10:54:47.380160Z",
    "end_time" : "2022-09-02T10:55:58.466602Z",
    "environment" : {
      "ort_version" : "DOCKER-SNAPSHOT",
      "java_version" : "11.0.16.1",
      "os" : "Linux",
      "processors" : 8,
      "max_memory" : 8334082048,
      "variables" : {
        "JAVA_HOME" : "/opt/java/openjdk",
        "ANDROID_HOME" : "/opt/android-sdk",
        "GOPATH" : "/tmp/go"
      },
      "tool_versions" : { }
    },
    "config" : {
      "allow_dynamic_versions" : false
    },
    "result" : {
      "projects" : [ {
        "id" : "Gradle:org.example:sample_gradle:1.0-SNAPSHOT",
        "definition_file_path" : "",
        "declared_licenses" : [ ],
        "declared_licenses_processed" : { },
        "vcs" : {
          "type" : "",
          "url" : "",
          "revision" : "",
          "path" : ""
        },
        "vcs_processed" : {
          "type" : "",
          "url" : "",
          "revision" : "",
          "path" : ""
        },
        "homepage_url" : "",
        "scope_names" : [ "runtimeClasspath", "testRuntimeClasspath" ]
      } ],
      "packages" : [ {
        "package" : {
          "id" : "Maven:com.codepoetics:protonpack:1.16",
          "purl" : "pkg:maven/com.codepoetics/protonpack@1.16",
          "authors" : [ "Dominic Fox" ],
          "declared_licenses" : [ "MIT" ],
          "declared_licenses_processed" : {
            "spdx_expression" : "MIT"
          },
          "description" : "Stream Utilities for Java 8",
          "homepage_url" : "https://github.com/poetix/protonpack",
          "binary_artifact" : {
            "url" : "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16.jar",
            "hash" : {
              "value" : "2f90f51a950cb68529613d077ef07194883a479a",
              "algorithm" : "SHA-1"
            }
          },
          "source_artifact" : {
            "url" : "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16-sources.jar",
            "hash" : {
              "value" : "3ddc16c880eefdf48354c1208cb12cce0e8189a7",
              "algorithm" : "SHA-1"
            }
          },
          "vcs" : {
            "type" : "Git",
            "url" : "git://github.com/poetix/protonpack.git",
            "revision" : "",
            "path" : ""
          },
          "vcs_processed" : {
            "type" : "Git",
            "url" : "https://github.com/poetix/protonpack.git",
            "revision" : "",
            "path" : ""
          }
        },
        "curations" : [ ]
      } ],
      "dependency_graphs" : {
        "Gradle" : {
          "packages" : [ "Maven:com.codepoetics:protonpack:1.16" ],
          "scopes" : {
            "org.example:sample_gradle:1.0-SNAPSHOT:runtimeClasspath" : [ {
              "root" : 0
            } ],
            "org.example:sample_gradle:1.0-SNAPSHOT:testRuntimeClasspath" : [ {
              "root" : 0
            } ]
          },
          "nodes" : [ { } ],
          "edges" : [ ]
        }
      },
      "has_issues" : false
    }
  },
  "scanner" : null,
  "advisor" : null,
  "evaluator" : null
}

Scanner:

{
  "repository" : {
    "vcs" : {
      "type" : "",
      "url" : "",
      "revision" : "",
      "path" : ""
    },
    "vcs_processed" : {
      "type" : "",
      "url" : "",
      "revision" : "",
      "path" : ""
    },
    "config" : { }
  },
  "analyzer" : {
    "start_time" : "2022-09-02T10:54:47.380160Z",
    "end_time" : "2022-09-02T10:55:58.466602Z",
    "environment" : {
      "ort_version" : "DOCKER-SNAPSHOT",
      "java_version" : "11.0.16.1",
      "os" : "Linux",
      "processors" : 8,
      "max_memory" : 8334082048,
      "variables" : {
        "JAVA_HOME" : "/opt/java/openjdk",
        "ANDROID_HOME" : "/opt/android-sdk",
        "GOPATH" : "/tmp/go"
      },
      "tool_versions" : { }
    },
    "config" : {
      "allow_dynamic_versions" : false
    },
    "result" : {
      "projects" : [ {
        "id" : "Gradle:org.example:sample_gradle:1.0-SNAPSHOT",
        "definition_file_path" : "",
        "declared_licenses" : [ ],
        "declared_licenses_processed" : { },
        "vcs" : {
          "type" : "",
          "url" : "",
          "revision" : "",
          "path" : ""
        },
        "vcs_processed" : {
          "type" : "",
          "url" : "",
          "revision" : "",
          "path" : ""
        },
        "homepage_url" : "",
        "scope_names" : [ "runtimeClasspath", "testRuntimeClasspath" ]
      } ],
      "packages" : [ {
        "package" : {
          "id" : "Maven:com.codepoetics:protonpack:1.16",
          "purl" : "pkg:maven/com.codepoetics/protonpack@1.16",
          "authors" : [ "Dominic Fox" ],
          "declared_licenses" : [ "MIT" ],
          "declared_licenses_processed" : {
            "spdx_expression" : "MIT"
          },
          "description" : "Stream Utilities for Java 8",
          "homepage_url" : "https://github.com/poetix/protonpack",
          "binary_artifact" : {
            "url" : "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16.jar",
            "hash" : {
              "value" : "2f90f51a950cb68529613d077ef07194883a479a",
              "algorithm" : "SHA-1"
            }
          },
          "source_artifact" : {
            "url" : "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16-sources.jar",
            "hash" : {
              "value" : "3ddc16c880eefdf48354c1208cb12cce0e8189a7",
              "algorithm" : "SHA-1"
            }
          },
          "vcs" : {
            "type" : "Git",
            "url" : "git://github.com/poetix/protonpack.git",
            "revision" : "",
            "path" : ""
          },
          "vcs_processed" : {
            "type" : "Git",
            "url" : "https://github.com/poetix/protonpack.git",
            "revision" : "",
            "path" : ""
          }
        },
        "curations" : [ ]
      } ],
      "dependency_graphs" : {
        "Gradle" : {
          "packages" : [ "Maven:com.codepoetics:protonpack:1.16" ],
          "scopes" : {
            "org.example:sample_gradle:1.0-SNAPSHOT:runtimeClasspath" : [ {
              "root" : 0
            } ],
            "org.example:sample_gradle:1.0-SNAPSHOT:testRuntimeClasspath" : [ {
              "root" : 0
            } ]
          },
          "nodes" : [ { } ],
          "edges" : [ ]
        }
      },
      "has_issues" : false
    }
  },
  "scanner" : {
    "start_time" : "2022-09-02T10:56:36.886056Z",
    "end_time" : "2022-09-02T10:56:46.198066Z",
    "environment" : {
      "ort_version" : "DOCKER-SNAPSHOT",
      "java_version" : "11.0.16.1",
      "os" : "Linux",
      "processors" : 8,
      "max_memory" : 8334082048,
      "variables" : {
        "JAVA_HOME" : "/opt/java/openjdk",
        "ANDROID_HOME" : "/opt/android-sdk",
        "GOPATH" : "/tmp/go"
      },
      "tool_versions" : { }
    },
    "config" : {
      "skip_concluded" : false,
      "create_missing_archives" : false,
      "detected_license_mapping" : {
        "LicenseRef-scancode-agpl-generic-additional-terms" : "NOASSERTION",
        "LicenseRef-scancode-generic-cla" : "NOASSERTION",
        "LicenseRef-scancode-generic-exception" : "NOASSERTION",
        "LicenseRef-scancode-generic-export-compliance" : "NOASSERTION",
        "LicenseRef-scancode-generic-tos" : "NOASSERTION",
        "LicenseRef-scancode-generic-trademark" : "NOASSERTION",
        "LicenseRef-scancode-gpl-generic-additional-terms" : "NOASSERTION",
        "LicenseRef-scancode-patent-disclaimer" : "NOASSERTION",
        "LicenseRef-scancode-warranty-disclaimer" : "NOASSERTION",
        "LicenseRef-scancode-other-copyleft" : "NOASSERTION",
        "LicenseRef-scancode-other-permissive" : "NOASSERTION",
        "LicenseRef-scancode-free-unknown" : "NOASSERTION",
        "LicenseRef-scancode-unknown" : "NOASSERTION",
        "LicenseRef-scancode-unknown-license-reference" : "NOASSERTION",
        "LicenseRef-scancode-unknown-spdx" : "NOASSERTION"
      },
      "ignore_patterns" : [ "**/*.ort.yml", "**/*.spdx.yml", "**/*.spdx.yaml", "**/*.spdx.json", "**/META-INF/DEPENDENCIES", "**/META-INF/DEPENDENCIES.txt", "**/META-INF/NOTICE", "**/META-INF/NOTICE.txt" ]
    },
    "results" : {
      "scan_results" : {
        "Gradle:org.example:sample_gradle:1.0-SNAPSHOT" : [ {
          "provenance" : { },
          "scanner" : {
            "name" : "ScanCode",
            "version" : "30.1.0",
            "configuration" : "--copyright --license --info --strip-root --timeout 300 --json-pp"
          },
          "summary" : {
            "start_time" : "2022-09-02T10:56:46.195626Z",
            "end_time" : "2022-09-02T10:56:46.195626Z",
            "package_verification_code" : "",
            "licenses" : [ ],
            "copyrights" : [ ],
            "issues" : [ {
              "timestamp" : "2022-09-02T10:56:46.196030Z",
              "source" : "ScanCode",
              "message" : "Could not download 'Gradle:org.example:sample_gradle:1.0-SNAPSHOT': DownloadException: Download failed for 'Gradle:org.example:sample_gradle:1.0-SNAPSHOT'.\nSuppressed: DownloadException: No VCS URL provided for 'Gradle:org.example:sample_gradle:1.0-SNAPSHOT'. Please make sure the published POM file includes the SCM connection, see: https://docs.gradle.org/current/userguide/publishing_maven.html#sec:modifying_the_generated_pom\nSuppressed: DownloadException: No source artifact URL provided for 'Gradle:org.example:sample_gradle:1.0-SNAPSHOT'.",
              "severity" : "ERROR"
            } ]
          }
        } ],
        "Maven:com.codepoetics:protonpack:1.16" : [ {
          "provenance" : {
            "source_artifact" : {
              "url" : "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16-sources.jar",
              "hash" : {
                "value" : "3ddc16c880eefdf48354c1208cb12cce0e8189a7",
                "algorithm" : "SHA-1"
              }
            }
          },
          "scanner" : {
            "name" : "ScanCode",
            "version" : "30.1.0",
            "configuration" : "--copyright --license --info --strip-root --timeout 300 --json-pp"
          },
          "summary" : {
            "start_time" : "2022-09-02T10:56:39.000561263Z",
            "end_time" : "2022-09-02T10:56:44.000595773Z",
            "package_verification_code" : "36969f5e9dde518b25766a15f28d293d0be4ec26",
            "licenses" : [ ],
            "copyrights" : [ ]
          }
        } ]
      },
      "storage_stats" : {
        "num_reads" : 3,
        "num_hits" : 0
      },
      "has_issues" : true
    }
  },
  "advisor" : null,
  "evaluator" : null
}

Evaluator:

---
repository:
  vcs:
    type: ""
    url: ""
    revision: ""
    path: ""
  vcs_processed:
    type: ""
    url: ""
    revision: ""
    path: ""
  config: {}
analyzer:
  start_time: "2022-09-02T10:54:47.380160Z"
  end_time: "2022-09-02T10:55:58.466602Z"
  environment:
    ort_version: "DOCKER-SNAPSHOT"
    java_version: "11.0.16.1"
    os: "Linux"
    processors: 8
    max_memory: 8334082048
    variables:
      JAVA_HOME: "/opt/java/openjdk"
      ANDROID_HOME: "/opt/android-sdk"
      GOPATH: "/tmp/go"
    tool_versions: {}
  config:
    allow_dynamic_versions: false
  result:
    projects:
    - id: "Gradle:org.example:sample_gradle:1.0-SNAPSHOT"
      definition_file_path: ""
      declared_licenses: []
      declared_licenses_processed: {}
      vcs:
        type: ""
        url: ""
        revision: ""
        path: ""
      vcs_processed:
        type: ""
        url: ""
        revision: ""
        path: ""
      homepage_url: ""
      scope_names:
      - "runtimeClasspath"
      - "testRuntimeClasspath"
    packages:
    - package:
        id: "Maven:com.codepoetics:protonpack:1.16"
        purl: "pkg:maven/com.codepoetics/protonpack@1.16"
        authors:
        - "Dominic Fox"
        declared_licenses:
        - "MIT"
        declared_licenses_processed:
          spdx_expression: "MIT"
        description: "Stream Utilities for Java 8"
        homepage_url: "https://github.com/poetix/protonpack"
        binary_artifact:
          url: "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16.jar"
          hash:
            value: "2f90f51a950cb68529613d077ef07194883a479a"
            algorithm: "SHA-1"
        source_artifact:
          url: "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16-sources.jar"
          hash:
            value: "3ddc16c880eefdf48354c1208cb12cce0e8189a7"
            algorithm: "SHA-1"
        vcs:
          type: "Git"
          url: "git://github.com/poetix/protonpack.git"
          revision: ""
          path: ""
        vcs_processed:
          type: "Git"
          url: "https://github.com/poetix/protonpack.git"
          revision: ""
          path: ""
      curations: []
    dependency_graphs:
      Gradle:
        packages:
        - "Maven:com.codepoetics:protonpack:1.16"
        scopes:
          org.example:sample_gradle:1.0-SNAPSHOT:runtimeClasspath:
          - root: 0
          org.example:sample_gradle:1.0-SNAPSHOT:testRuntimeClasspath:
          - root: 0
        nodes:
        - {}
        edges: []
    has_issues: false
scanner:
  start_time: "2022-09-02T10:56:36.886056Z"
  end_time: "2022-09-02T10:56:46.198066Z"
  environment:
    ort_version: "DOCKER-SNAPSHOT"
    java_version: "11.0.16.1"
    os: "Linux"
    processors: 8
    max_memory: 8334082048
    variables:
      JAVA_HOME: "/opt/java/openjdk"
      ANDROID_HOME: "/opt/android-sdk"
      GOPATH: "/tmp/go"
    tool_versions: {}
  config:
    skip_concluded: false
    create_missing_archives: false
    detected_license_mapping:
      LicenseRef-scancode-agpl-generic-additional-terms: "NOASSERTION"
      LicenseRef-scancode-generic-cla: "NOASSERTION"
      LicenseRef-scancode-generic-exception: "NOASSERTION"
      LicenseRef-scancode-generic-export-compliance: "NOASSERTION"
      LicenseRef-scancode-generic-tos: "NOASSERTION"
      LicenseRef-scancode-generic-trademark: "NOASSERTION"
      LicenseRef-scancode-gpl-generic-additional-terms: "NOASSERTION"
      LicenseRef-scancode-patent-disclaimer: "NOASSERTION"
      LicenseRef-scancode-warranty-disclaimer: "NOASSERTION"
      LicenseRef-scancode-other-copyleft: "NOASSERTION"
      LicenseRef-scancode-other-permissive: "NOASSERTION"
      LicenseRef-scancode-free-unknown: "NOASSERTION"
      LicenseRef-scancode-unknown: "NOASSERTION"
      LicenseRef-scancode-unknown-license-reference: "NOASSERTION"
      LicenseRef-scancode-unknown-spdx: "NOASSERTION"
    ignore_patterns:
    - "**/*.ort.yml"
    - "**/*.spdx.yml"
    - "**/*.spdx.yaml"
    - "**/*.spdx.json"
    - "**/META-INF/DEPENDENCIES"
    - "**/META-INF/DEPENDENCIES.txt"
    - "**/META-INF/NOTICE"
    - "**/META-INF/NOTICE.txt"
  results:
    scan_results:
      Gradle:org.example:sample_gradle:1.0-SNAPSHOT:
      - provenance: {}
        scanner:
          name: "ScanCode"
          version: "30.1.0"
          configuration: "--copyright --license --info --strip-root --timeout 300\
            \ --json-pp"
        summary:
          start_time: "2022-09-02T10:56:46.195626Z"
          end_time: "2022-09-02T10:56:46.195626Z"
          package_verification_code: ""
          licenses: []
          copyrights: []
          issues:
          - timestamp: "2022-09-02T10:56:46.196030Z"
            source: "ScanCode"
            message: "Could not download 'Gradle:org.example:sample_gradle:1.0-SNAPSHOT':\
              \ DownloadException: Download failed for 'Gradle:org.example:sample_gradle:1.0-SNAPSHOT'.\n\
              Suppressed: DownloadException: No VCS URL provided for 'Gradle:org.example:sample_gradle:1.0-SNAPSHOT'.\
              \ Please make sure the published POM file includes the SCM connection,\
              \ see: https://docs.gradle.org/current/userguide/publishing_maven.html#sec:modifying_the_generated_pom\n\
              Suppressed: DownloadException: No source artifact URL provided for 'Gradle:org.example:sample_gradle:1.0-SNAPSHOT'."
            severity: "ERROR"
      Maven:com.codepoetics:protonpack:1.16:
      - provenance:
          source_artifact:
            url: "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16-sources.jar"
            hash:
              value: "3ddc16c880eefdf48354c1208cb12cce0e8189a7"
              algorithm: "SHA-1"
        scanner:
          name: "ScanCode"
          version: "30.1.0"
          configuration: "--copyright --license --info --strip-root --timeout 300\
            \ --json-pp"
        summary:
          start_time: "2022-09-02T10:56:39.000561263Z"
          end_time: "2022-09-02T10:56:44.000595773Z"
          package_verification_code: "36969f5e9dde518b25766a15f28d293d0be4ec26"
          licenses: []
          copyrights: []
    storage_stats:
      num_reads: 3
      num_hits: 0
    has_issues: true
advisor: null
evaluator:
  start_time: "2022-09-02T10:57:28.795782Z"
  end_time: "2022-09-02T10:57:36.955998Z"
  violations: []

Reporter (NOTICE)

    The applicable license information is listed below:

                ----

                Package:              com.codepoetics:protonpack:1.16
                    No Source Artifact available

            The following copyrights and licenses were found in the source code of this package:

            No Copyrights available

        Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

I am using this customized ftl template:

[#--
    Copyright (C) 2020 HERE Europe B.V.
    Copyright (C) 2021 Bosch.IO GmbH
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at
        http://www.apache.org/licenses/LICENSE-2.0
    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
    SPDX-License-Identifier: Apache-2.0
    License-Filename: LICENSE
--]
[#--
    The notice file generated by this template consists of the following sections:
    * The licenses and associated copyrights for all projects merged into a single list.
    * The archived license files, licenses and associated copyrights for dependencies listed by package.
    Excluded projects and packages are ignored.
--]
[#assign noticeCategoryName = "include-in-notice-file"]
[#-- Add the licenses of the projects. --]
[#assign hasNoticeProjectLicenses = false]
[#--
    Merge the licenses and copyrights of all projects into a single list. The default LicenseView.ALL is used because
    projects cannot have a concluded license (compare with the handling of packages below).
--]
[#assign mergedLicenses = helper.mergeLicenses(projects)]
[#-- Filter for those licenses that are categorized to be included in notice files. --]
[#assign filteredLicenses = helper.filterForCategory(mergedLicenses, noticeCategoryName)]
[#list filteredLicenses as resolvedLicense]
    [#assign licenseName = resolvedLicense.license.simpleLicense()]
    [#assign licenseText = licenseTextProvider.getLicenseText(licenseName)!]
    [#if !licenseText?has_content][#continue][/#if]
    [#if !hasNoticeProjectLicenses]
        This software includes external packages and source code.
        The applicable license information is listed below:

        ----
        [#assign hasNoticeProjectLicenses = true]
    [#else]
        --
    [/#if]
    [#assign copyrights = resolvedLicense.getCopyrights()]
    [#if copyrights?has_content]

    [/#if]
    ${copyrights?join("\n", "", "\n")}
    ${licenseText}
    [#assign exceptionName = resolvedLicense.license.exception()!]
    [#assign exceptionText = licenseTextProvider.getLicenseText(exceptionName)!]
    [#if exceptionText?has_content]
        ${exceptionText}
    [/#if]
[/#list]
[#-- Add the licenses of all packages. --]
[#if packages?has_content]
    [#if hasNoticeProjectLicenses]
        ----

    [/#if]
    This software depends on external packages and source code.
    The applicable license information is listed below:

[/#if]
[#list packages?filter(p -> !p.excluded) as package]
    [#assign hasNoticePackageLicenses = false]
[#-- List the content of archived license files and associated copyrights. --]
    [#list package.licenseFiles.files as licenseFile]
        [#if !hasNoticePackageLicenses]
            ----

            Package: [#if package.id.namespace?has_content]${package.id.namespace}:[/#if]${package.id.name}:${package.id.version}
            [#assign hasNoticePackageLicenses = true]
        [/#if]

        This package contains the file ${licenseFile.path} with the following contents:

        ${licenseFile.text}
        [#assign copyrights = licenseFile.getCopyrights()]
        [#if copyrights?has_content]
            The following copyright holder information relates to the license(s) above:

            ${copyrights?join("\n", "")}
        [/#if]
    [/#list]
[#--
    Filter the licenses of the package using LicenseView.ONLY_DECLARED. Also filter all licenses that are configured not to be included in
    notice files, and filter all licenses that are contained in the license files already printed above.
--]
    [#assign licensesFilteredBySource = LicenseView.ONLY_DECLARED.filter(package.license.licenses)]
    [#assign resolvedLicenses = helper.filterForCategory(
    package.licensesNotInLicenseFiles(licensesFilteredBySource),
    noticeCategoryName
    )]
    [#assign isFirst = true]
    [#list resolvedLicenses as resolvedLicense]
        [#assign licenseName = resolvedLicense.license.simpleLicense()]
        [#assign licenseLocations = resolvedLicense.locations]
        [#assign licenseText = licenseTextProvider.getLicenseText(licenseName)!]
        [#if !licenseText?has_content][#continue][/#if]
        [#if isFirst]
            [#if !hasNoticePackageLicenses]
                ----

                Package: [#if package.id.namespace?has_content]             ${package.id.namespace}:[/#if]${package.id.name}:${package.id.version}
                [#assign hasNoticePackageLicenses = true]
                [#if licenseLocations?has_content && licenseLocations[0].provenance?has_content]
                    [#assign artifactProvenance = licenseLocations[0].provenance]
                    [#if artifactProvenance?has_content && artifactProvenance.sourceArtifact?has_content]

                        Source Artifact:    ${artifactProvenance.sourceArtifact.url}
                    [#elseif artifactProvenance?has_content && artifactProvenance.binaryArtifact?has_content]
                        Source Artifact (Binary):    ${artifactProvenance.binaryArtifact.url}
                    [#elseif artifactProvenance?has_content && artifactProvenance.vcsInfo?has_content]
                        Source Artifact (VCS Info):    ${artifactProvenance.vcsInfo.url}
                    [#else]
                        @TODO: Check for other Source Artifact Category
                    [/#if]
                [#else]
                    No Source Artifact available
                [/#if]
            [/#if]

            The following copyrights and licenses were found in the source code of this package:

            [#assign isFirst = true]
        [#else]
            --
        [/#if]
        [#assign copyrights = resolvedLicense.getCopyrights()]
        [#if copyrights?has_content]
            ${copyrights?join("\n", "", "\n")}
        [#else]
            No Copyrights available
        [/#if]

        ${licenseText}

        [#assign exceptionName = resolvedLicense.license.exception()!]
        [#assign exceptionText = licenseTextProvider.getLicenseText(exceptionName)!]
        [#if exceptionText?has_content]
            License Exceptions:

            ${exceptionText}
        [/#if]
    [/#list]
[/#list]

As ort.conf it is used:

ort {
  licenseFilePatterns {
    licenseFilenames = ["license*"]
    patentFilenames = [patents]
    rootLicenseFilenames = ["readme*"]
  }

  severeIssueThreshold = ERROR
  severeRuleViolationThreshold = ERROR

  enableRepositoryPackageCurations = true
  enableRepositoryPackageConfigurations = true

  analyzer {
    allowDynamicVersions = true
  }

  downloader {
    allowMovingRevisions = true

    # Only used by the CLI tool when the '--license-classifications-file' option is specified.
      includedLicenseCategories = [
        copyleft,
        copyleft-provide-sourcecode,
        weak-copyleft,
        weak-copyleft-provide-sourcecode,
        proprietary,
        permissive,
        public-domain,
        no-assertion,
        not-for-commercial-use,
        include-in-notice-file,
        include-source-code-offer-in-notice-file
      ]

    sourceCodeOrigins = [
      VCS, ARTIFACT
    ]
  }

  scanner {
    skipConcluded = true

    archive {
      enabled = false

      postgresStorage {
        connection {
          url = ${POSTGRES_URL}
            schema = "public"
            username = "ort"
            password = ${POSTGRES_PASSWORD}
            sslmode = "disable"
            parallelTransactions = 5
        }
      }
    }

    createMissingArchives = false

    storages {

      postgres {
        connection {
          url = ${POSTGRES_URL}
          schema = "public"
          username = "ort"
          password = ${POSTGRES_PASSWORD}
          sslmode = "disable"
        }
      }
    }

    storageReaders = [
      postgres
    ]

    storageWriters = [
      postgres
    ]
  }
}

@sschuberth : Do you have an explanation why I do not have the information in the reporter even though it could be sucessfully retreived in analyzer, scanner and evaluator step?

schvvarzekatze commented 2 years ago

I tried to fix it manually with this curation in the curations.yml:

- id: "Maven:com.codepoetics:protonpack:1.16"
  curations:
    binary_artifact:
      url: "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16.jar"
      hash: "2f90f51a950cb68529613d077ef07194883a479a"
      hash_algorithm: "SHA-1"
    source_artifact:
      url: "https://repo.maven.apache.org/maven2/com/codepoetics/protonpack/1.16/protonpack-1.16-sources.jar"
      hash: "3ddc16c880eefdf48354c1208cb12cce0e8189a7"
      hash_algorithm: "SHA-1"

The evaulator was run with --package-curations-file "/project/ort/curations.yml"

But also this did not help to fix the issue. Do you have an idea what could be the reason?

schvvarzekatze commented 2 years ago

resolved by: https://github.com/oss-review-toolkit/ort/issues/5902