Update ScanCode Toolkit version

clearlydefined / crawler

A service that crawls projects and packages for information relevant to ClearlyDefined

MIT License

48 stars 33 forks source link

Update ScanCode Toolkit version #502

Open qtomlinson opened 1 year ago

qtomlinson commented 1 year ago

In the scancode result, warning is reported:

"OUTDATED": "WARNING: Outdated ScanCode Toolkit version! You are using an outdated version of ScanCode Toolkit: 30.1.0 released on: 2021-09-24. A new version is available with important improvements including bug and security fixes, updated license, copyright and package detection, and improved scanning accuracy. Please download and install the latest version of ScanCode. Visit https://github.com/nexB/scancode-toolkit/releases for details."

elrayle commented 1 year ago

ClearlyDefined/crawler is 2 major versions behind on nexB/scancode-toolkit (referred to generically as scancode). There have been output format changes in at least one of the updates. ClearlyDefined/service processes raw tool output and needs to be updated to work with the latest scancode output format.

Related Work

qtomlinson commented 11 months ago

Changes needed: Crawler:

Command-line options in config/cdConfig.js
Dockerfile and DevDockerfile

Service:

providers/summary/scancode.js needs to be modified to extract the ScanCode output to the summary.

Test cases:

Cases in point 1. These licenses have been verified.
Another interesting case to cover is when the license from the package summary differs from the license from the root files. For example, composer/packagist/mmucklo/krumo/0.7.0 (see definitionServiceTest).

Other considerations:

Memory requirements for the newer version compared to the version in use. Is any adjustment to the configuration of our infrastructure necessary?

lumaxis commented 11 months ago

Thank you @qtomlinson! I started looking into this and will report back with any findings 🙂

qtomlinson commented 11 months ago

@lumaxis For cases where the license of the package differs from the license of the root files (e.g., composer/packagist/mmucklo/krumo/0.7.0), it is interesting to discover the new behavior, which may be different or better than what we currently have.