Open dabutvin opened 5 years ago
@dabutvin sorry for the late reply. ScanCode returns detected licenses as expressions. You get these either:
withe the license_expressions
attribute a files
object. This is a list of license expressions strings found in a given file. If you want to get the composite expression you could wrap them in parens and AND
them together.
Alternatively the licenses
attribute lists each license together with the corresponding license matched_rules
license_expression
attribute which is a single string.
In both cases the expressions are made of ScanCode license keys (which are the keys in https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses)
If you want instead SPDX ids each license
object that has one has an spdx_license_key
attribute: so you could parse the expression and then replace the key for SPDX IDs (and possibly use SPDX LicenseRef-... for license keys that are not know from SPDX).
Alternatively, we could update ScanCode to return also license_expressions that would be made of SPDX keys.
FWIW, expressions are reasonably common in the JQuery world. Many Rust crates have such expressions. And there are several other cases: about ~20% of the 7K rules in ScanCode return expressions with more than one license
so in any case AND'ing license objects is unlikely to be correct as you are missing the OR and WITH cases
Thanks @pombredanne I'm looking into fixing this now, should I expect a significant difference in scancode 3.0?
@pombredanne @dabutvin Just checking on this for our upgrade to Scancode 3.0?
so this has come a long way, but we still have one gap. The following explains the origin of the problem, not sure what the solution should be.
Here is the repo we are using here for reference https://github.com/rust-lang/regex It is meant to be licensed as "MIT OR Apache-2.0"
With the latest scancode we get the right license on a file by file basis
[
path: 'src/backtrack.rs',
license: 'MIT OR Apache-2.0',
attributions: ['Copyright 2014-2015 The Rust Project'],
hashes: { sha1: 'd40e188e9bd713fd1b858b914bae098b320d2799' }
},
{
path: 'src/compile.rs',
license: 'MIT OR Apache-2.0',
attributions: ['Copyright 2014-2016 The Rust Project'],
hashes: { sha1: '222404078d2e3eca8f79aa5ae2be4fb4cdf11c6f' }
},
where we get it wrong is declaring the license
we are getting
{ licensed: { declared: 'Apache-2.0 AND MIT' },
To try and declare a license, first, we check the scancode_output.summary.packages[0].declared_license
, but it is empty
"summary": {
"license_expressions": [
{ "value": "mit OR apache-2.0", "count": 55 },
{ "value": null, "count": 53 },
{ "value": "apache-2.0 OR mit", "count": 2 },
{ "value": "apache-2.0", "count": 1 },
{ "value": "mit", "count": 1 },
{ "value": "mit-synopsys", "count": 1 }
],
"copyrights": [{ "value": null, "count": 58 }, { "value": "Copyright (c) The Rust Project", "count": 28 }],
"holders": [{ "value": null, "count": 58 }, { "value": "The Rust Project", "count": 28 }],
"authors": [{ "value": null, "count": 84 }, { "value": "The Rust Project", "count": 2 }],
"programming_language": [
{ "value": "Rust", "count": 69 },
{ "value": null, "count": 14 },
{ "value": "Objective-C", "count": 2 },
{ "value": "Python", "count": 1 }
],
"packages": []
},
So we try to declare the license by reading the "is_license_text": true
file propery
In this case, there are 2 files marked as "is_license_text": true
LICENSE-APACHE
and LICENSE-MIT
each with individual licenses in them.
We take these licenses and AND them together to declare a license.
_getLicenseByIsLicenseText(files) {
const fullLicenses = files
.filter(file => file.is_license_text && file.licenses)
.reduce((licenses, file) => {
file.licenses.forEach(license => {
licenses.add(this._createExpressionFromLicense(license))
})
return licenses
}, new Set())
return this._joinExpressions(fullLicenses)
}
this gives us Apache-2.0 AND MIT
this is still an issue with the summarizer
the readme is correctly detecting MIT OR Apache-2.0 but our summarizer is joining them with ANDs for the declared
Finally I got some good input for you guys, but it will require us to decide how to handle things.
The thing is, our summarizer for Scancode will look into the files array provided by the tool, and not in the (very smart) content.summary.license_expressions. Going file-by-file, the system finds that there are both Apache and MIT identifiable licenses, and ANDs them together by default. As seen here.
On to the decision-making part.
For reference, the package mentioned gives from Scancode the following summarized counts and values (the winner with 117 is the mix we are suggested to take into account):
[{"value":null,"count":159},{"value":"mit OR apache-2.0","count":117},{"value":"public-domain","count":16},{"value":"apache-2.0 OR mit","count":5},{"value":"apache-2.0","count":1},{"value":"mit","count":1},{"value":"mit-synopsys","count":1},{"value":"unknown","count":1}]
Still, how dangerous is it to put it to file count? What if we got a number of equal-file-count license types?
All comments appreciated.
The component's license is MIT OR Apache-2.0 and not MIT AND Apache-2.0 as initially reported for https://clearlydefined.io/definitions/crate/cratesio/-/regex/1.0.6 It is, however, MIT AND Apache-2.0 for https://clearlydefined.io/definitions/git/github/rust-lang/regex/18a71d0a30a6dcdcd86d1af6dd9cb0688b89f2ee. scancode 3.2.2 was run for both of the components. Any idea why the declared license is different? As to is it dangerous or not, we would need to analyze tens or hundreds of sample packages to be sure. We probably do not want to invest time in that right now.
@geneh I am looking at the example you give with cratesio, but initially the harvested info is a lot more complex and for the older version also includes earlier version runs of ScanCode, which may be an explanation (even though I will debug this to make sure this is the case).
@geneh what would you say if we remove the original data, and recrawl? We would lose existing information from ScanCode, but I think that is exactly what is wrong. Just an idea.
@ignacionr Do you mean recrawl all the components? Why would the outcome be any different if the scanocde version is the same? Do you mean we should recompute definitions for the affected scancode results?
I just ran scancode on the most recent version of regex and saved the result as a gist. Please note that scancode has different results for...
"license_expressions": ["mit OR apache-2.0" ]
"license_expressions": ["mit OR apache-2.0", "unicode"]
Compare this to the code snippet above from scancode.js which calls _joinExpressions (that function always combines with AND
) for all the Full license files. And thus we would expect that result to be "mit AND apache-2.0 AND unicode", but it is only "mit AND apache-2.0" because the Unicode is only referenced in the README (the full Unicode license is not in the top level directory, but rather in regex-syntax/src/unicode_tables/LICENSE-UNICODE
).
Because the resulting package "regex" contains a compilation of Unicode data and Rust source code the likely correct SPDX expression for the ensemble is "(mit OR apache-2.0) AND unicode". NOTE: the parentheses are required because otherwise the AND
takes precedence, per 4) Order of Precedence and Parentheses in the SPDX spec.
So here are the questions:
@jeffmcaffer @pombredanne @iamwillbar Would love to hear your opinions on this one.
When we are gathering up the
file.licenses.license
orfile.licenses.spdx_license_key
data we default to AND them together and this is not always correct.Given this rust crate
We list these discovered files as
Apache-2.0 and MIT
, but this is only because scancode found both and we AND them together.The scancode output has other information we should probably consume for example:
cc @pombredanne