Is the unmapped state still an error failing the scanner job?

[woznik]

Hello I recently discovered that on the final report for one nodejs project there are Issues reported by ORT scanner saying that there are unmapped license found

Like: PyPI::click:7.1.2 | The declared license 'BSD License' could not be mapped to a valid license or parsed as an SPDX expression.

in the past these kind of status failed the scanner job and I had to put the license remapping in the curations.yml file

Doen it still work like this?

I ask because I have another scan where in the log I see:

Found 6 errors, 0 warnings, 0 hints. There are issues with a severity equal to or greater than the WARNING threshold.

In the logs there are no errors, only 6 WARNINGS

Is this issue WARNING -> ERROR related to this:

00:38:40.589 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download 'NPM::make-fetch-happen:5.0.2' sources to '/tmp/ort-ScanCode16308538202973995538/NPM/unknown/make-fetch-happen/5.0.2' from VCS...
00:38:40.589 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Using processed VcsInfo(type=Git, url=https://github.com/zkat/make-fetch-happen.git, revision=b317c90d4d03367b6dd84ee676b850c11721887f, path=). Original was VcsInfo(type=Git, url=git+https://github.com/zkat/make-fetch-happen.git, revision=b317c90d4d03367b6dd84ee676b850c11721887f, path=).
00:38:40.589 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Detected VCS type 'Git' from type name 'Git'.
00:38:40.933 [main] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Adding Git revision 'b317c90d4d03367b6dd84ee676b850c11721887f' (taken from package meta-data) as a candidate.
00:38:41.206 [main] INFO  org.ossreviewtoolkit.downloader.vcs.Git - No Git revision for package 'make-fetch-happen' and version '5.0.2' found: IOException: No matching tag found for version '5.0.2' among tags 'v1.3.1', 'v1.4.0', 'v1.5.0', 'v1.2.0', 'v1.2.1', 'v1.3.0', 'v1.5.1', 'v1.6.0', 'v1.7.0', 'v2.4.6', 'v2.4.5', 'v2.4.8', 'v2.4.7', 'v1.0.0', 'v1.0.1', 'v1.1.0', 'v2.4.9', 'v2.0.1', 'v2.1.0', 'v2.2.0', 'v2.0.2', 'v2.0.3', 'v2.2.2', 'v2.4.0', 'v2.0.4', 'v2.2.1', 'v2.3.0', 'v2.2.4', 'v2.4.2', 'v2.6.0', 'v2.2.3', 'v2.4.1', 'v2.5.0', 'v2.2.6', 'v2.4.4', 'v2.2.5', 'v2.4.3', 'v2.0.0', 'v2.4.13', 'v2.4.12', 'v2.4.11', 'v2.4.10', 'v3.0.0', 'v4.0.0', 'v4.0.1'. Please create a tag whose name contains the version.
00:38:41.206 [main] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Trying revision candidate 'b317c90d4d03367b6dd84ee676b850c11721887f' (1 of 1)...
00:38:41.206 [main] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Trying to fetch only revision 'b317c90d4d03367b6dd84ee676b850c11721887f' with depth limited to 50.
00:38:41.208 [main] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'git fetch --depth 50 origin b317c90d4d03367b6dd84ee676b850c11721887f' in '/tmp/ort-ScanCode16308538202973995538/NPM/unknown/make-fetch-happen/5.0.2'...
00:38:41.721 [main] WARN  org.ossreviewtoolkit.downloader.vcs.Git - Could not fetch only revision 'b317c90d4d03367b6dd84ee676b850c11721887f': IOException: Running 'git fetch --depth 50 origin b317c90d4d03367b6dd84ee676b850c11721887f' in '/tmp/ort-ScanCode16308538202973995538/NPM/unknown/make-fetch-happen/5.0.2' failed with exit code 128:
fatal: remote error: upload-pack: not our ref b317c90d4d03367b6dd84ee676b850c11721887f

or it is still a unmapped case?

Thank you in advance

[sschuberth]

There are multiple issues that come into play here:

The declared license 'BSD License' could not be mapped

We used to have a mapping for 'BSD License' but decided to remove it due to ambiguity.

I had to put the license remapping in the curations.yml file

Yes, the correct way to address this kind of issue is to use a package-specific declared license mapping in a curation.

In the logs there are no errors, only 6 WARNINGS

There is a difference between log output and recorded ORT issues. Log output in general might also come from third-party libraries used by ORT (if they use the same logger as ORT). The issue statistics refer to ORT issues in the ORT result files.

[woznik]

Ok, thanks I will apply the curations.yml file but the still it does not fully resolve my doubts because in my first NodeJS project I had unmapped cases and the scanner job wasn't failed - isseus were reported in the HTML report, no curation was required (runnning analyse, scan, report in gitlab pipeline job)

[sschuberth]

I forgot to mention one additional thing that comes into play: There were (and still are) some inconsistencies about what counts as an issue (and at what severity), exit codes, reporter visualization, etc. Also see https://github.com/oss-review-toolkit/ort/issues/4096. Also note the new severeIssueThreshold configuration variable. In any case, please try with the very latest ORT commit (and configure severeIssueThreshold to your needs). Things should have been improved.

[sschuberth]

Also https://github.com/oss-review-toolkit/ort/issues/3324 seems to be related.

[woznik]

ok, this a vital information what counts as an issue, thank you very much Senastian

[woznik]

Could you share what are the other things that count as an issue? At my end I have remapped all unmapped licenses but still the job fails No errors either. The scan-results.yml is 18MB so I cannot place it here

[sschuberth]

Could you share what are the other things that count as an issue?

AFAIK, unmapped licenses are the only "non-issue thing" that count as issues (or better: issues are dynamically created but not serialized as issues, as explained here).

At my end I have remapped all unmapped licenses but still the job fails

Which ORT commit are you using?

The scan-results.yml is 18MB so I cannot place it here

Not even if you ZIP it?

[woznik]

Hello I use the commit: 82cf932674669918b97d336854df383d7474b2ec (5 days ago)

the issues which I tracked in the scan-results files are:

issues:
          - timestamp: "2021-06-23T01:38:23.591564Z"
            source: "ScanCode"
            message: "ERROR: Timeout after 300 seconds while scanning file 'third_party/dotnet/nunit-3.6.0/net-2.0/nunit.framework.tests.dll'."
            severity: "ERROR"
          - timestamp: "2021-06-23T01:38:23.591601Z"
            source: "ScanCode"
            message: "ERROR: Timeout after 300 seconds while scanning file 'third_party/dotnet/nunit-3.6.0/net-3.5/nunit.framework.tests.dll'."
            severity: "ERROR"

Yes, a zip file is better scan-result.zip

[sschuberth]

So, some general remarks about my observations of the scan result:

• Note that each section (from the analyzer, advisor, scanner etc.) should have it's own has_issues flag in the result file.
• There is no has_issues flag at all in the analyzer part of your result file, which might be a regression from our switch to a new dependency graph format. We'll investigate. Edit: This was fixed by https://github.com/oss-review-toolkit/ort/pull/4220.
• The scanner section has has_issues: true, which is correct as there are issues in the form of scan timeouts.
• Issue resolutions do not affect the serialized has_issues flag, as the resolutions are only applied as a "view" on top of the results by the reporter.

[sschuberth]

And to finally directly answer your original question:

Is the unmapped state still an error failing the scanner job?

Unmapped licenses never failed the scanner step, but only the analyzer step. And whether unmapped licenses fail the analyzer step now actually depends on the configured severeIssueThreshold (which defaults to Severity.WARNING): Unmapped license issues are of Severity.HINT if a concluded license has been set, and of Severity.WARNING otherwise. So, if you have not set a concluded license for a package, and you have unmapped licenses in that package, then unmapped licenses will fail the analyzer by default.

[woznik]

The scanner section has has_issues: true, which is correct as there are issues in the form of scan timeouts.

Is there a way to resolve these kind of issues? I don't want to change the timeout in the scancode settings because it affects all my previously cached scanned packages (for nodejs it was 1200 -> 4h of processing)

Documentation says:

Resolutions set in resolution.yml file are only taken into account by the reporter, while the analyzer and scanner ignore them.

In my pipeline the scanner ends with exit code 1 so I cannot use resilution.yml file here

Unmapped licenses never failed the scanner step (...) unmapped licenses will fail the analyzer by default. From my observation the pipeline: analyzer, scanner, reporter is failed on scanner step? When I implement curation to umapped license then all three steps go well

[sschuberth]

Is there a way to resolve these kind of issues?

You basically gave yourself the answer already: These issues can be resolved (that is, what ORT calls "resolved") by writing issue resolutions. But having those issue resolutions does not affect the exit code of the scanner. This is a valid request, though, and I have filed https://github.com/oss-review-toolkit/ort/issues/4222 about it (I thought we already had an issue about this, but apparently that was not the case).

So, does that answer your original question? Are you good to close this issue in favor of the other existing / created ones?

[woznik]

yes of course I am closing the issue