search

oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.61k stars 312 forks source link

FileProvenanceFileStorage cannot find file listings #7041

Closed MarcelBochtler closed 2 months ago

MarcelBochtler commented 1 year ago

Since https://github.com/oss-review-toolkit/ort/pull/6970, we're seeing the following error in our scans:

ERROR org.ossreviewtoolkit.model.utils.FileProvenanceFileStorage - Could not read from 66eab51c0401c5041aa975dbf9c112cecfb9dfdd/file_list.xz: FileNotFoundException: /root/.ort/scanner/file-lists/66eab51c0401c5041aa975dbf9c112cecfb9dfdd/file_list.xz (No such file or directory)

We did not yet configure the postgres storage for the file listings, but from my understanding not configuring that, shouldn't lead to errors in the scan.

Edit: We're using ORT's legacy docker image for these scans.

sschuberth commented 1 year ago

@MarcelBochtler can this still be reproduced after the recent PRs #7119, #7121 and #7129 in that area?

MarcelBochtler commented 1 year ago

In the meantime we configured Postgres for the file lists, which solves this issue.

As a test I disabled it again, and yes, this can still be reproduced.

alpianon commented 8 months ago

I'm experiencing the same bug and using same workaround. ORT 19.1.0, built from sources with OpenJDK 11 on Ubuntu 22.04

sschuberth commented 8 months ago

In any case, glad you're taking a look at ORT, @alpianon. Note that BitBake-related stuff is still in a fork only. Please ensure to ask question rather sooner than later, e.g. in Slack, as we're aware that ORT is not trivial to setup and configure, esp. with performance in mind.

alpianon commented 8 months ago

In any case, glad you're taking a look at ORT, @alpianon. Note that BitBake-related stuff is still in a fork only. Please ensure to ask question rather sooner than later, e.g. in Slack, as we're aware that ORT is not trivial to setup and configure, esp. with performance in mind.

I'm not working only on bitbake-based projects, now I have a java+npm project to scan :)

Glad that you started developing a bitbake analyzer. Maybe we can talk about it on slack

sschuberth commented 8 months ago

Maybe we can talk about it on slack

We should absolutely talk! But I guess a dedicated meeting makes sense. I'll reach out to you privately.

rettichschnidi commented 5 months ago

I am seeing this error too when doing the steps described in the tutorial:

$ ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results/
 ______________________________
/        \_______   \__    ___/ The OSS Review Toolkit, version 22.0.0-049.sha.ed7fdbe,
|    |   | |       _/ |    |    built with JDK 11.0.22+7, running under Java 17.0.11.
|    |   | |    |   \ |    |    Executing 'scan' as 'username' on Linux
\________/ |____|___/ |____|    with 24 CPUs and a maximum of 24128 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = /home/username/.ort/config
ORT_DATA_DIR = /home/username/.ort
HOME = /home/username
SHELL = /bin/bash
TERM = xterm-256color
GOPATH = /home/username/opt/go

Looking for ORT configuration in the following file:
        /home/username/.ort/config/config.yml (does not exist)

Scanning projects with:
        ScanCode (version 32.1.0)
Scanning packages with:
        ScanCode (version 32.1.0)
01:24:09.973 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.model.utils.FileProvenanceFileStorage - Could not read from 39f76b06552989aaa626341d3c3936645bad36b3/file_list: FileNotFoundException: /home/username/.ort/scanner/file-lists/39f76b06552989aaa626341d3c3936645bad36b3/file_list (No such file or directory)
Caused by: FileNotFoundException: /home/username/.ort/scanner/file-lists/39f76b06552989aaa626341d3c3936645bad36b3/file_list.xz (No such file or directory)
01:24:31.163 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.model.utils.FileProvenanceFileStorage - Could not read from c643dce0af603a22b7be53afb0844ce302fa93e6/file_list: FileNotFoundException: /home/username/.ort/scanner/file-lists/c643dce0af603a22b7be53afb0844ce302fa93e6/file_list (No such file or directory)
Caused by: FileNotFoundException: /home/username/.ort/scanner/file-lists/c643dce0af603a22b7be53afb0844ce302fa93e6/file_list.xz (No such file or directory)
Wrote scan result to '/home/username/some-path/ort-example-mime-types/mime-types-ort-scanner-results/scan-result.yml' (0.42 MiB) in 133.975115ms.
The scan took 39.348319185s.
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 0 errors, 0 warnings, 0 hints.

But when running the same command a 2nd time the errors are gone, as the mentioned files have been created during the first run:

$ ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results/
 ______________________________
/        \_______   \__    ___/ The OSS Review Toolkit, version 22.0.0-049.sha.ed7fdbe,
|    |   | |       _/ |    |    built with JDK 11.0.22+7, running under Java 17.0.11.
|    |   | |    |   \ |    |    Executing 'scan' as 'username' on Linux
\________/ |____|___/ |____|    with 24 CPUs and a maximum of 24128 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = /home/username/.ort/config
ORT_DATA_DIR = /home/username/.ort
HOME = /home/username
SHELL = /bin/bash
TERM = xterm-256color
GOPATH = /home/username/opt/go

Looking for ORT configuration in the following file:
        /home/username/.ort/config/config.yml (does not exist)

Scanning projects with:
        ScanCode (version 32.1.0)
Scanning packages with:
        ScanCode (version 32.1.0)
Wrote scan result to '/home/username/some-path/ort-example-mime-types/mime-types-ort-scanner-results/scan-result.yml' (0.42 MiB) in 209.344576ms.
The scan took 3.591540482s.
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 0 errors, 0 warnings, 0 hints.
sschuberth commented 5 months ago

Are there any files present below /home/username/.ort/scanner/file-lists? I'm curious whether files or only being looked up under the wrong location maybe, or whether they're not being created in the first place.

rettichschnidi commented 5 months ago

Are there any files present below /home/username/.ort/scanner/file-lists?

Yes, but not the ones ort scan was not able to find. Those files however exist after ort scan ran (and complained about them being missed).

I can reproduce the error like this: 1) rm -rf ~/.ort/ 2) ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results

Running ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results a 2nd time will then be much faster and does no longer show this error.

reuvenperetz commented 2 months ago

Hi, is there any update about this issue? I'm experiencing the same problem...

mnonnenmacher commented 2 months ago

The error message was a false positive. It was not properly checked if the file list archive exists before trying to download it.

When the archive does not exist yet, it will be created, so no error message should be logged in this case.