Closed schvvarzekatze closed 1 week ago
sschuberth
commented
1 month ago Some questions:
test/fixtures/testdir5 directory?
schvvarzekatze
commented
1 month ago I have added parts of the necessary information. Unfortunately the tmp directory is cleared as soon as the job ist processed. So we will retry the run with a sleep.
sschuberth
commented
1 month ago I've created https://github.com/oss-review-toolkit/ort/pull/8989 to help with debugging this issue. Can you run again with that PR merged?
schvvarzekatze
commented
1 month ago Thank you then I will rebuild a docker image as soon as that PR is merged and rerun the ort scan.
schvvarzekatze
commented
2 weeks ago Hello @sschuberth: I have rerun it with the logging you provided. I hope this helps to clarify.
08:20:14.380 [DefaultDispatcher-worker-16] INFO org.eclipse.jgit.lib.Repository - Updating working tree from https://github.com/babel/babel.git.
08:20:14.380 [DefaultDispatcher-worker-16] INFO org.ossreviewtoolkit.plugins.versioncontrolsystems.git.Git - Trying to fetch only revision 'a7620bd266ae1345975767bbc7abf09034437017' with depth limited to 50.
08:20:16.544 [DefaultDispatcher-worker-16] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git checkout a7620bd266ae1345975767bbc7abf09034437017' in '/tmp/ort-DefaultWorkingTreeCache17766079121285209300'...
08:20:24.021 [DefaultDispatcher-worker-6] INFO org.eclipse.jgit.lib.Repository - Updating working tree from https://github.com/babel/babel.git.
08:20:24.021 [DefaultDispatcher-worker-6] INFO org.ossreviewtoolkit.plugins.versioncontrolsystems.git.Git - Trying to fetch only revision 'e0375121bfea96d277f49543dc543f116523cfd0' with depth limited to 50.
08:20:25.729 [DefaultDispatcher-worker-6] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git checkout e0375121bfea96d277f49543dc543f116523cfd0' in '/tmp/ort-DefaultWorkingTreeCache17766079121285209300'...
08:20:32.877 [DefaultDispatcher-worker-7] INFO org.eclipse.jgit.lib.Repository - Updating working tree from https://github.com/babel/babel.git.
08:20:32.877 [DefaultDispatcher-worker-7] INFO org.ossreviewtoolkit.plugins.versioncontrolsystems.git.Git - Trying to fetch only revision '38c23cded40af3ccc8d2c39dbe165e9b446e55a3' with depth limited to 50.
08:20:34.590 [DefaultDispatcher-worker-7] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git checkout 38c23cded40af3ccc8d2c39dbe165e9b446e55a3' in '/tmp/ort-DefaultWorkingTreeCache17766079121285209300'...
08:25:17.405 [DefaultDispatcher-worker-11] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git checkout f069547c3d25cb9602fdbc1d9eb1f34046dc8353' in '/tmp/ort-DefaultWorkingTreeCache141016[2549](https://git.intra.xyz.com/xyz/project/-/jobs/1486812#L2549)2012631433'...
08:28:44.515 [main] INFO org.ossreviewtoolkit.scanner.Scanner - Created file lists for 1069 provenances in 12m 44.782219038s.
08:28:44.544 [main] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@18026052 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
08:28:44.545 [main] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@6e068ac9 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
08:28:44.545 [main] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@3d0352 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
08:28:44.545 [main] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@bb6f3f7 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
08:28:44.545 [main] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@388f1258 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
08:28:46.123 [main] INFO org.ossreviewtoolkit.scanner.Scanner - Creating file archives for 2 package(s).
08:28:46.125 [main] INFO org.eclipse.jgit.lib.Repository - Updating working tree from https://github.com/feross/safe-buffer.git.
08:28:46.125 [main] INFO org.ossreviewtoolkit.plugins.versioncontrolsystems.git.Git - Trying to fetch only revision 'e4d9032c56faa8a024dce8b501b4e31d5c2ab300' with depth limited to 50.
08:28:46.614 [main] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git checkout e4d9032c56faa8a024dce8b501b4e31d5c2ab300' in '/tmp/ort-DefaultWorkingTreeCache12522413385701667413'...
08:28:46.856 [main] INFO org.ossreviewtoolkit.model.utils.FileArchiver - Archiving files matching [license*, patents, readme*] from '/tmp/ort-DefaultProvenanceDownloader13860422904187057827'...
08:28:46.859 [main] INFO org.ossreviewtoolkit.model.utils.FileArchiver - Archived directory '/tmp/ort-DefaultProvenanceDownloader13860422904187057827' in 2.297392ms.
08:28:46.887 [main] INFO org.ossreviewtoolkit.model.utils.FileArchiver - Wrote archive of directory '/tmp/ort-DefaultProvenanceDownloader13860422904187057827' to storage in 27.907610ms.
08:28:46.891 [main] INFO org.eclipse.jgit.lib.Repository - Updating working tree from https://github.com/isaacs/node-tar.git.
08:28:46.891 [main] INFO org.ossreviewtoolkit.plugins.versioncontrolsystems.git.Git - Trying to fetch only revision 'bef7b1e4ffab822681fea2a9b22187192ed14717' with depth limited to 50.
08:28:48.378 [main] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git checkout bef7b1e4ffab822681fea2a9b22187192ed14717' in '/tmp/ort-DefaultWorkingTreeCache11332046505647301986'...
08:28:48.740 [main] INFO org.ossreviewtoolkit.model.utils.FileArchiver - Archiving files matching [license*, patents, readme*] from '/tmp/ort-DefaultProvenanceDownloader2483837061893383424'...
08:28:48.755 [main] INFO org.ossreviewtoolkit.model.utils.FileArchiver - Archived directory '/tmp/ort-DefaultProvenanceDownloader2483837061893383424' in 13.718638ms.
08:28:48.772 [main] INFO org.ossreviewtoolkit.model.utils.FileArchiver - Wrote archive of directory '/tmp/ort-DefaultProvenanceDownloader2483837061893383424' to storage in 16.965873ms.
08:28:48.795 [main] INFO org.ossreviewtoolkit.scanner.Scanner - Created file archives for 2 package(s) in 2.671656413s.
08:28:50.090 [main] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'scancode --version' in '/home/ort'...
08:28:51.913 [main] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'scancode --version' in '/home/ort'...
Exception in thread "main" java.io.IOException: Could not delete file '/tmp/ort-DefaultWorkingTreeCache593982492075894716/test/fixtures/testdir5'.
at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively(Extensions.kt:151)
at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively$default(Extensions.kt:128)
at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively(Extensions.kt:132)
at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively$default(Extensions.kt:128)
at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively(Extensions.kt:132)
at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively$default(Extensions.kt:128)
at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively(Extensions.kt:132)
at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively$default(Extensions.kt:128)
at org.ossreviewtoolkit.downloader.DefaultWorkingTreeCache.shutdown(WorkingTreeCache.kt:99)
at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand$runScanners$5.invokeSuspend(ScannerCommand.kt:240)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.runScanners(ScannerCommand.kt:240)
at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.run(ScannerCommand.kt:142)
at com.github.ajalt.clikt.parsers.Parser.finalizeAndRun(Parser.kt:348)
at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:218)
at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:245)
at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:42)
at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:457)
at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:454)
at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:474)
at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:481)
at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:85)
Uploading artifacts for failed job
00:03
Uploading artifacts...
build/oss-review-toolkit/*: found 4 matching artifact files and directories
Uploading artifacts as "archive" to coordinator... 201 Created id=1486812 responseStatus=201 Created token=glcbt-64
Cleaning up project directory and file based variables
00:02
ERROR: Job failed: exit code 1@schvvarzekatze I don't see any of the new debug logging in there. Note that you explicitly have to run ORT with --debug to enable these.
schvvarzekatze
commented
2 weeks ago Thank you. I tried it another time with --debug
run-ort --info scan -i "/xyz-project/${ortDirFromProjectRoot}/analyzer/analyzer-result.yml" -o "/xyz-project/${ortDirFromProjectRoot}/scan/" --skip-excluded || test -f "$scanOutput" --debugbut the logs still look the same.
I do not find
Deleting cached working tree for in the full log history.
sschuberth
commented
2 weeks ago The --debug needs to replace the --info you have at the beginning:
run-ort --info scan ...I just tried it but with a local run (as all logging results could not be collected on gitlab). The error is the same anyway. The logs are in the initial comment.
sschuberth
commented
1 week ago Note to myself: The new debug logs now indicate that the problem probably is the non-UTF8 filename in https://github.com/fshost/node-dir/tree/master/test/fixtures/testdir5.
schvvarzekatze
commented
1 week ago Thank you, with the fix the scan runs sucessfully again.
Describe the bug
The scan step exits with error after file archives for all packages are created.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
All steps are processed sucessfully and reports generated.
Console / log output
Add console and / or log output that shows the error and additional context. No screenshots of plain text please, to keep text searchable.
Environment
Output of the
ort requirementscommand:Or manually specify:
And specify (relevant parts of) your ORT configuration (
config.yml):Additional context
Add any other context about the problem here.