oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.61k stars 313 forks source link

Scan step crashes with SVNException #9291

Open schvvarzekatze opened 1 month ago

schvvarzekatze commented 1 month ago

When using

For Gradle packages the scan crashes with an error

The same config including environment is only working and generation successful reports for Yarn as package manager.

Question: Are there special config changes to be considered to make the recent ORT version run as well for Gradle in the scan step for libraries from Subversion as verson control system type? Does it help to exclude packages like this or could exceptions be caught?

To Reproduce

Steps to reproduce the behavior:

  1. Create flattened project with all dependencies
  2. Start ORT run with analyzer and scan for Package Manager Gradle

Expected behavior

All relevant libraries determined from the analyzer result are scanned

Console / log output

Logs

03:07:38.539 [main] INFO  org.ossreviewtoolkit.scanner.Scanner - Scanning RepositoryProvenance(vcsInfo=VcsInfo(type=Subversion, url=http://svn.apache.org/repos/asf/xml/commons, revision=tags/xml-commons-resolver-1_2, path=), resolvedRevision=1905005) (89 of 387)...
Exception in thread "main" java.io.IOException: Unable to initialize a Subversion working tree in '/tmp/ort-DefaultWorkingTreeCache10536384807322232179' from http://svn.apache.org/repos/asf/xml/commons.
    at org.ossreviewtoolkit.plugins.versioncontrolsystems.subversion.Subversion.initWorkingTree(Subversion.kt:100)
    at org.ossreviewtoolkit.downloader.DefaultWorkingTreeCache.getWorkingTree(WorkingTreeCache.kt:85)
    at org.ossreviewtoolkit.downloader.DefaultWorkingTreeCache.use(WorkingTreeCache.kt:65)
    at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.downloadFromVcs(ProvenanceDownloader.kt:104)
    at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.access$downloadFromVcs(ProvenanceDownloader.kt:80)
    at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader$download$1.invokeSuspend(ProvenanceDownloader.kt:96)
    at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader$download$1.invoke(ProvenanceDownloader.kt)
    at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader$download$1.invoke(ProvenanceDownloader.kt)
    at org.ossreviewtoolkit.utils.ort.UtilsKt$runBlocking$1.invokeSuspend(Utils.kt:231)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:263)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at org.ossreviewtoolkit.utils.ort.UtilsKt.runBlocking(Utils.kt:231)
    at org.ossreviewtoolkit.utils.ort.UtilsKt.runBlocking$default(Utils.kt:229)
    at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.download(ProvenanceDownloader.kt:96)
    at org.ossreviewtoolkit.scanner.Scanner.scanPath(Scanner.kt:568)
    at org.ossreviewtoolkit.scanner.Scanner.runPathScanners(Scanner.kt:445)
    at org.ossreviewtoolkit.scanner.Scanner.scan(Scanner.kt:177)
    at org.ossreviewtoolkit.scanner.Scanner$scan$3.invokeSuspend(Scanner.kt)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:263)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:47)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
    at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.runScanners(ScannerCommand.kt:236)
    at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.run(ScannerCommand.kt:142)
    at com.github.ajalt.clikt.core.CoreCliktCommandKt.parse(CoreCliktCommand.kt:107)
    at com.github.ajalt.clikt.core.CoreCliktCommandKt.main(CoreCliktCommand.kt:78)
    at com.github.ajalt.clikt.core.CoreCliktCommandKt.main(CoreCliktCommand.kt:90)
    at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:85)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E170000: URL 'http://svn.apache.org/repos/asf/xml/commons' doesn't exist
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:852)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
    at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
    at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
    at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
    at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777)
    at org.ossreviewtoolkit.plugins.versioncontrolsystems.subversion.Subversion.initWorkingTree(Subversion.kt:89)
    ... 35 more

Environment

And specify (relevant parts of) your ORT configuration (config.yml):

ort:
  forceOverwrite: false
  addAuthorsToCopyrights: true
  enableRepositoryPackageCurations: true 
  enableRepositoryPackageConfigurations: true

  licenseFilePatterns:
    licenseFilenames: [ 'license*' ]
    patentFilenames: [ patents ]
    rootLicenseFilenames: [ 'readme*' ]

  severeIssueThreshold: ERROR
  severeRuleViolationThreshold: ERROR

  analyzer:
    allowDynamicVersions: true
    enabledPackageManagers: [ GradleInspector, Yarn, NPM ]

  downloader:
    allowMovingRevisions: true
    # Only used if the '--license-classifications-file' option is specified.
    includedLicenseCategories:
      - copyleft
      - copyleft-provide-sourcecode
      - weak-copyleft
      - weak-copyleft-provide-sourcecode
      - proprietary
      - permissive
      - public-domain
      - no-assertion
      - not-for-commercial-use
      - include-in-notice-file
      - include-source-code-offer-in-notice-file

  scanner:
    skipConcluded: true

    archive:

      postgresStorage:
        connection:
          url: ${POSTGRES_URL}
          schema: public
          username: ort
          password: ${POSTGRES_PASSWORD}
          sslmode: disable

    createMissingArchives: true

    storages:
      clearlyDefined:
        serverUrl: 'https://api.clearlydefined.io'

      postgres:
        connection:
          url: ${POSTGRES_URL}
          schema: public
          username: ort
          password: ${POSTGRES_PASSWORD}
          sslmode: disable

    storageReaders: [ postgres, clearlyDefined ]
    storageWriters: [ postgres ]

    provenanceStorage:

      postgresStorage:
        connection:
          url: ${POSTGRES_URL}
          schema: public
          username: ort
          password: ${POSTGRES_PASSWORD}
          sslmode: disable

Additional context

cf. Description above

Thank you for more information.

sschuberth commented 1 month ago

This probably has the same root cause as https://github.com/oss-review-toolkit/ort/issues/5232 and is not related to Gradle specifically.