AyanSinhaMahapatra
commented
2 years ago @DennisClark this is already fixed (in the sense we don't return an unknown anymore) in the LicenseDetection branch for the upcoming release: https://github.com/nexB/scancode-toolkit/tree/add-license-detection.
Similar to Issue 2 in https://github.com/nexB/scancode-toolkit/issues/3069#issuecomment-1237003830 and also similar to this issue reported by eclipse foundation here: https://github.com/nexB/scancode-toolkit/issues/2878#issuecomment-1128612554, this is solved by:
Here the detection rule is "unknown-intro-followed-by-match" i.e. an unknown intro was there followed by a proper detection and so this unknown can be removed. This is achieved by tagging specific rules as is_license_intro as True.
It would be great if SCTK could be improved to interpret these instances of a dual-license choice being declared
But this still detects (gpl-2.0 OR apache-2.0) and not just apache-2.0 as marked by Doris chooses Apache 2.0 License or similar statements.
To solve that part we do have to add rules, and this cannot be solved by generalized processing.
New license detection looks like this:
"detected_license_expression": "bsd-new AND ((gpl-2.0 OR apache-2.0) AND bsd-new)",
"detected_license_expression_spdx": "BSD-3-Clause AND ((GPL-2.0-only OR Apache-2.0) AND BSD-3-Clause)",
"license_detections": [
{
"license_expression": "bsd-new",
"detection_rules": [
"not-combined"
],
"matches": [
{
"score": 100.0,
"start_line": 1,
"end_line": 1,
"matched_length": 3,
"match_coverage": 100.0,
"matcher": "2-aho",
"license_expression": "bsd-new",
"rule_identifier": "bsd-new_308.RULE",
"referenced_filenames": [],
"is_license_text": false,
"is_license_notice": false,
"is_license_reference": true,
"is_license_tag": false,
"is_license_intro": false,
"rule_length": 3,
"rule_relevance": 100,
"matched_text": "3-clause BSD",
"licenses": [
{
"key": "bsd-new",
"name": "BSD-3-Clause",
"short_name": "BSD-3-Clause",
"category": "Permissive",
"is_exception": false,
"is_unknown": false,
"owner": "Regents of the University of California",
"homepage_url": "http://www.opensource.org/licenses/BSD-3-Clause",
"text_url": "http://www.opensource.org/licenses/BSD-3-Clause",
"reference_url": "https://scancode-licensedb.aboutcode.org/bsd-new",
"scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-new.LICENSE",
"scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-new.yml",
"spdx_license_key": "BSD-3-Clause",
"spdx_url": "https://spdx.org/licenses/BSD-3-Clause"
}
]
}
]
},
{
"license_expression": "(gpl-2.0 OR apache-2.0) AND bsd-new",
"detection_rules": [
"unknown-intro-followed-by-match"
],
"matches": [
{
"score": 100.0,
"start_line": 4,
"end_line": 5,
"matched_length": 3,
"match_coverage": 100.0,
"matcher": "2-aho",
"license_expression": "unknown-license-reference",
"rule_identifier": "lead-in_unknown_30.RULE",
"referenced_filenames": [],
"is_license_text": false,
"is_license_notice": false,
"is_license_reference": false,
"is_license_tag": false,
"is_license_intro": true,
"rule_length": 3,
"rule_relevance": 100,
"matched_text": "dual-licensed\n under",
"licenses": [
{
"key": "unknown-license-reference",
"name": "Unknown License file reference",
"short_name": "Unknown License reference",
"category": "Unstated License",
"is_exception": false,
"is_unknown": true,
"owner": "Unspecified",
"homepage_url": null,
"text_url": "",
"reference_url": "https://scancode-licensedb.aboutcode.org/unknown-license-reference",
"scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/unknown-license-reference.LICENSE",
"scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/unknown-license-reference.yml",
"spdx_license_key": "LicenseRef-scancode-unknown-license-reference",
"spdx_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/unknown-license-reference.LICENSE"
}
]
},
{
"score": 33.33,
"start_line": 4,
"end_line": 6,
"matched_length": 11,
"match_coverage": 33.33,
"matcher": "3-seq",
"license_expression": "gpl-2.0 OR apache-2.0",
"rule_identifier": "gpl-2.0_or_apache-2.0_2.RULE",
"referenced_filenames": [
"COPYING",
"LICENSE.Apache"
],
"is_license_text": false,
"is_license_notice": true,
"is_license_reference": false,
"is_license_tag": false,
"is_license_intro": false,
"rule_length": 33,
"rule_relevance": 100,
"matched_text": "licensed\n under both the GPLv2 and Apache 2.0 License. We select Apache 2.0\nLicense.",
"licenses": [
{
"key": "gpl-2.0",
"name": "GNU General Public License 2.0",
"short_name": "GPL 2.0",
"category": "Copyleft",
"is_exception": false,
"is_unknown": false,
"owner": "Free Software Foundation (FSF)",
"homepage_url": "http://www.gnu.org/licenses/gpl-2.0.html",
"text_url": "http://www.gnu.org/licenses/gpl-2.0.txt",
"reference_url": "https://scancode-licensedb.aboutcode.org/gpl-2.0",
"scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/gpl-2.0.LICENSE",
"scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/gpl-2.0.yml",
"spdx_license_key": "GPL-2.0-only",
"spdx_url": "https://spdx.org/licenses/GPL-2.0-only"
},
{
"key": "apache-2.0",
"name": "Apache License 2.0",
"short_name": "Apache 2.0",
"category": "Permissive",
"is_exception": false,
"is_unknown": false,
"owner": "Apache Software Foundation",
"homepage_url": "http://www.apache.org/licenses/",
"text_url": "http://www.apache.org/licenses/LICENSE-2.0",
"reference_url": "https://scancode-licensedb.aboutcode.org/apache-2.0",
"scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.LICENSE",
"scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.yml",
"spdx_license_key": "Apache-2.0",
"spdx_url": "https://spdx.org/licenses/Apache-2.0"
}
]
},
{
"score": 36.36,
"start_line": 10,
"end_line": 11,
"matched_length": 12,
"match_coverage": 36.36,
"matcher": "3-seq",
"license_expression": "gpl-2.0 OR apache-2.0",
"rule_identifier": "gpl-2.0_or_apache-2.0_2.RULE",
"referenced_filenames": [
"COPYING",
"LICENSE.Apache"
],
"is_license_text": false,
"is_license_notice": true,
"is_license_reference": false,
"is_license_tag": false,
"is_license_intro": false,
"rule_length": 33,
"rule_relevance": 100,
"matched_text": "code is licensed under both GPLv2 and Apache 2.0 License.\n Doris chooses Apache 2.0 License.",
"licenses": [
{
"key": "gpl-2.0",
"name": "GNU General Public License 2.0",
"short_name": "GPL 2.0",
"category": "Copyleft",
"is_exception": false,
"is_unknown": false,
"owner": "Free Software Foundation (FSF)",
"homepage_url": "http://www.gnu.org/licenses/gpl-2.0.html",
"text_url": "http://www.gnu.org/licenses/gpl-2.0.txt",
"reference_url": "https://scancode-licensedb.aboutcode.org/gpl-2.0",
"scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/gpl-2.0.LICENSE",
"scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/gpl-2.0.yml",
"spdx_license_key": "GPL-2.0-only",
"spdx_url": "https://spdx.org/licenses/GPL-2.0-only"
},
{
"key": "apache-2.0",
"name": "Apache License 2.0",
"short_name": "Apache 2.0",
"category": "Permissive",
"is_exception": false,
"is_unknown": false,
"owner": "Apache Software Foundation",
"homepage_url": "http://www.apache.org/licenses/",
"text_url": "http://www.apache.org/licenses/LICENSE-2.0",
"reference_url": "https://scancode-licensedb.aboutcode.org/apache-2.0",
"scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.LICENSE",
"scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.yml",
"spdx_license_key": "Apache-2.0",
"spdx_url": "https://spdx.org/licenses/Apache-2.0"
}
]
},
{
"score": 100.0,
"start_line": 14,
"end_line": 16,
"matched_length": 33,
"match_coverage": 100.0,
"matcher": "2-aho",
"license_expression": "gpl-2.0 OR apache-2.0",
"rule_identifier": "gpl-2.0_or_apache-2.0_2.RULE",
"referenced_filenames": [
"COPYING",
"LICENSE.Apache"
],
"is_license_text": false,
"is_license_notice": true,
"is_license_reference": false,
"is_license_tag": false,
"is_license_intro": false,
"rule_length": 33,
"rule_relevance": 100,
"matched_text": "This source code is licensed under both the GPLv2 (found in the\n COPYING file in the root directory) and Apache 2.0 License\n (found in the LICENSE.Apache file in the root directory).",
"licenses": [
{
"key": "gpl-2.0",
"name": "GNU General Public License 2.0",
"short_name": "GPL 2.0",
"category": "Copyleft",
"is_exception": false,
"is_unknown": false,
"owner": "Free Software Foundation (FSF)",
"homepage_url": "http://www.gnu.org/licenses/gpl-2.0.html",
"text_url": "http://www.gnu.org/licenses/gpl-2.0.txt",
"reference_url": "https://scancode-licensedb.aboutcode.org/gpl-2.0",
"scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/gpl-2.0.LICENSE",
"scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/gpl-2.0.yml",
"spdx_license_key": "GPL-2.0-only",
"spdx_url": "https://spdx.org/licenses/GPL-2.0-only"
},
{
"key": "apache-2.0",
"name": "Apache License 2.0",
"short_name": "Apache 2.0",
"category": "Permissive",
"is_exception": false,
"is_unknown": false,
"owner": "Apache Software Foundation",
"homepage_url": "http://www.apache.org/licenses/",
"text_url": "http://www.apache.org/licenses/LICENSE-2.0",
"reference_url": "https://scancode-licensedb.aboutcode.org/apache-2.0",
"scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.LICENSE",
"scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.yml",
"spdx_license_key": "Apache-2.0",
"spdx_url": "https://spdx.org/licenses/Apache-2.0"
}
]
},
{
"score": 95.0,
"start_line": 19,
"end_line": 20,
"matched_length": 19,
"match_coverage": 100.0,
"matcher": "2-aho",
"license_expression": "bsd-new",
"rule_identifier": "bsd-new_1169.RULE",
"referenced_filenames": [
"LICENSE"
],
"is_license_text": false,
"is_license_notice": true,
"is_license_reference": false,
"is_license_tag": false,
"is_license_intro": false,
"rule_length": 19,
"rule_relevance": 95,
"matched_text": "Use of this source code is governed by a BSD-style license that can be\n found in the LICENSE file.",
"licenses": [
{
"key": "bsd-new",
"name": "BSD-3-Clause",
"short_name": "BSD-3-Clause",
"category": "Permissive",
"is_exception": false,
"is_unknown": false,
"owner": "Regents of the University of California",
"homepage_url": "http://www.opensource.org/licenses/BSD-3-Clause",
"text_url": "http://www.opensource.org/licenses/BSD-3-Clause",
"reference_url": "https://scancode-licensedb.aboutcode.org/bsd-new",
"scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-new.LICENSE",
"scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-new.yml",
"spdx_license_key": "BSD-3-Clause",
"spdx_url": "https://spdx.org/licenses/BSD-3-Clause"
}
]
}
]
}
],
"license_clues": [],There was also a bug related to how we group matches into LicenseDetection, I have solved this to factor in license intros when doing this grouping.
Here are the scan results for you to look at:
Old scan just this issue: doris-issue-3082.json.txt
New scan just this issue: doris-add-license-detection-issue-3082.json.txt
Old scan entire file: doris-v31.1.1-LICENSE-dist.json.txt
New scan entire file: doris-add-license-detection-LICENSE-dist.json.txt
There are still a few unknown-license-reference remaining after this which are a result of lines like these:
ANTLR 4 Runtime -- licenses/LICENSE-antlr4.txt
- org.antlr:antlr4-runtime:4.7 (http://www.antlr.org/antlr4-runtime)
* Automaton -- licenses/LICENSE-automaton.txt
- dk.brics.automaton:automaton:1.11-8 (http://www.brics.dk/automaton/)
* JLine -- licenses/LICENSE-jline.txt
- jline:jline:2.12 (http://nexus.sonatype.org/oss-repository-hosting.html/jline)But these are also beyond what we can achieve by generalized processing, but they can be solved by rules.
scanned doris-1.1.1-rc03 ( available at https://github.com/apache/doris/archive/refs/tags/1.1.1-rc03.tar.gz ) using scancode-toolkit-31.0.2 and although it detected most of the licenses in the rather complex notice (attached) in doris-1.1.1-rc03/dist/LICENSE-dist.txt it returns unknown-license-reference twice for portions of the following text:
See lines 9463 forward in the scan results for multiple cases where SCTK finds multiple cases of both unknown-license-reference and also identifies the license choices.
In these cases the authors declare the choice of Apache 2.0, using slightly different language in each case. It would be great if SCTK could be improved to interpret these instances of a dual-license choice being declared in the license notice and not to return an unknown-license-reference.
LICENSE-dist.txt.zip
doris-1.1.1-rc03-results.json.zip