License Report Generator Notes

[doki-nordic]

Place for notes.

[doki-nordic]

Methods of publishing scancode-toolkit-cache.json

1. Publish as a github action atifact:
   • our script must use Github API to download latest artifact
2. Publish to the same server as documentation.
   • if using existsting method, it can be published at: https://developer.nordicsemi.com/nRF_Connect_SDK_dev/doc/west-sbom/scancode-toolkit-cache.json
   • if file cannot be in the "doc" directory, modification of monitor scripts are required
3. Publish as a commit in master branch
   • it will create a commit once a day
   • no need to download from internet by the user - it will be already in the repository
4. As a commit in seperate repository:
   • what/where repository?

SITE NOTE:

Database can contain timestamp when it was created and when update has been checked last time. The script will check for updates when, e.g.: creation_time > 25h AND last_check_time > 24h (assuming that database is created once a day)

[doki-nordic]

Planned CI use cases:

• [ ] Create and publish scancode-toolkit-cache.json.
• [ ] Check if all files contained in the PR have valid SPDX tag.
• [ ] Run check for entire NCS if there is no regression regarding licenses.
• [ ] Any more?

Other CI related:

• [ ] Add github action (runs e.g. once a week) that checks if SPDX license list is up to date and creates a PR with updated version if needed.
• [ ] Run nightly script that refreshes the cache database file and publishes it.

[doki-nordic]

To do:

• [x] ncs/nrf/samples/bluetooth/peripheral_lbs$ west sbom -d build/ --license-detectors spdx-tag @doki-nordic Exception. file: ```input_build.py line 100 Exception: "ar -t "build/modules/nrf/drivers/hw_cc310/lib..nrfdrivers__hw_cc310.a"" command exited with error code 9
• [x] Case insensitive spdx license id. @doki-nordic
• [x] Add -n parameter @maje-emb
• [x] Add relative west top dir as path in database (helper function in common.py) @maje-emb
• [x] Improve database detector @maje-emb
• [x] Rename scancode-cache to cache @maje-emb
• [x] Display warning when list of input files is empty @doki-nordic
• [x] Use west logger @doki-nordic @maje-emb
• [x] Add own exception @doki-nordic
• [x] Add exception handling for scancode-toolkit (common.py) @doki-nordic
• [x] Fix relative path returned by input-build @doki-nordic
• [x] For the scancode-toolkit detector the logs are messed. It is due to the scancode prints the detection information on stderr. @maje-emb
• [x] Check PEP8 standard in source code @maje-emb
• [x] Source code documentation @doki-nordic @maje-emb
• [x] Mark build-input as EXPERIMENTAL
• [x] Fix: Close file handles in command_execute in some cases.
• [x] Show help when no parameters are passed
• [x] Remove old (dead) code
• [x] Allow scancode from diffrent location
• [x] Test scancode installation with "--version" argument

[doki-nordic]

To do after PR:

• [ ] Add URLs to license exceptions if expression contains "WITH" operator
• [x] Verify why scancode-toolkit does not always returns valid SDPX identifier?

[doki-nordic]

To do:

Current PR

• [x] Doc: We should add sample of the report (e.g. screenshot) with some explanation @doki-nordic Replaced by overview in the documentation and maybe later: https://doki-nordic.github.io/test-demo-docs/sbom_report.html
• [x] Doc: Add information in the general license information and remove west spdx tool information @maje-emb
• [x] Doc: Tell how we are generating the list of files from the application build. (from user perspective) @doki-nordic
• [x] Doc: Add note that build must be successful and up to date, i.e. you have to rebuild it on each change of kconfig option. @doki-nordic
• [x] Finalize current file extraction from a build directory. @doki-nordic
• [x] Change the name to ncs-sbom
• [x] Add SPDX output template and verify its correctness with SPDX verification tools @maje-emb
• [x] Implement external file detector
• [x] Change UUID in the custom licenses from the database to west-ncs-sbom- prefix
• [x] Add special handling for libgcc.a,libc_nano.a,libm_nano.a - they are in the map file and not in the build system. Should be in the report
• [x] Add note to the report that some licenses were detected by scancode-toolkit with link to it
• [x] Implement map file parsing and verification @doki-nordic
• [x] Use spdx_license_key from scancode output if possible
• [x] Use license information (name and URL) from scancode output if possible
• [x] Use Nordic colors in the report
• [x] Use default build directory (if exists) if there is no input provided
• [x] Show clickable (in linux and vcs terminal) link to the output
• [x] Add dark and visual studio code CSS themes.
• [x] Fix scancode toolkit for Nordic license @maje-emb

Later - may be after NCS 2.0

• [ ] Implement extracting origin of each file:
  • repository
  • commit and a version tag if available
  • information if it was modified
  • In SPDX each repository can be mapped into packages
  • In HTML each license details section can be divided on more sections for each repository
• [ ] Run the script on entire repository and detect other problems and report problems to people responsible or fix them
• [ ] Run the script on entire repository to compare results between scancode-toolkit and other detectors
• [ ] Run the final script on every sample from nrf and zephyr repository
• [ ] Create workflow for PR checks. Help from testers needed.
  • Add workflow also to nrfxlib, decision point: will the nrfxlib have its own license-allow-list.yaml?
• [ ] Test for the correctness of the script
• [ ] Implement cache in some temporary directory (e.g. Zephyr's cache: USER_CACHE_DIR)
• [ ] Test for entire repository
• [ ] Add URLs to license exceptions if expression contains "WITH" operator
• [ ] Add information (maybe some symbol) that specific file was detected with external tool: scancode-toolkit.
• [ ] Maybe consider adding more information for each file as a CSS popup:
  • List of detectors
  • Hash
  • Source of the file (repository and commit/tag)

[github-actions[bot]]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.