search

aboutcode-org / scancode-toolkit

:mag: ScanCode detects licenses, copyrights, dependencies by "scanning code" ... to discover and inventory open source and third-party packages used in your code. Sponsored by NLnet project https://nlnet.nl/project/vulnerabilitydatabase, the Google Summer of Code, Azure credits, nexB and others generous sponsors!
https://github.com/aboutcode-org/scancode-toolkit/releases/
2.11k stars 545 forks source link

Improve `--cyclonedx` output option #2987

Open DennisClark opened 2 years ago

DennisClark commented 2 years ago

Running scancode-toolkit-31.0.0b5 on MacOS 11.6.6 The --json-pp output option works fine, but the --cyclonedx output option fails.

scancode-toolkit-31.0.0b5 % ./scancode --license --copyright --cyclonedx scancode_results_cyclonedx.json samples
Setup plugins... Collect file inventory... Scan files for: licenses, copyrights with 1 process(es)... [####################] 66
ERROR: failed to run output plugin: cyclonedx: Traceback (most recent call last): File "//scancode-toolkit-31.0.0b5/src/scancode/cli.py", line 1067, in run_codebase_plugins plugin.process_codebase(codebase, **kwargs) File "//scancode-toolkit-31.0.0b5/src/formattedcode/output_cyclonedx.py", line 735, in process_codebase bom = CycloneDxBom.from_codebase(codebase) File "//scancode-toolkit-31.0.0b5/src/formattedcode/output_cyclonedx.py", line 633, in from_codebase packages = codebase.attributes.packages AttributeError: 'CodebaseAttributes' object has no attribute 'packages'

Scanning done. Summary: licenses, copyrights with 1 process(es) Errors count: 0 Scan Speed: 1.12 files/sec. Initial counts: 44 resource(s): 33 file(s) and 11 directorie(s) Final counts: 44 resource(s): 33 file(s) and 11 directorie(s) Timings: scan_start: 2022-06-08T180404.780531 scan_end: 2022-06-08T180436.257832 setup_scan:licenses: 2.00s setup: 2.00s scan: 29.42s total: 31.49s Removing temporary files...done.

pombredanne commented 2 years ago

@DennisClark can you try with "--package" ? It should not crash anyway.

DennisClark commented 2 years ago

Running it with ./scancode --license --copyright --package --cyclonedx scancode_results_cyclonedx.json samples works without failing. About to review the results.

DennisClark commented 2 years ago

I got beyond the crash by specifying the --package option. Now there is a rather different problem which might be called "lost in translation". The samples provided for testing with sctk did not produce really interesting results, so I extracted a folder from `libX11-1.7.2.tar.bz2' and scanned that. The results using the --json-pp output option are extensive and detailed, but running the same scan with the --cyclonedx output option produces a very limited file with hardly any content. Associated files attached. Here are my commands:

./scancode --license --copyright --package --cyclonedx libX11-1.7.2-scan-cyclonedx.json /Users/me/Downloads/libX11-1.7.2 and ./scancode --license --copyright --package --json-pp libX11-1.7.2-scan-json-pp.json /Users/me/Downloads/libX11-1.7.2

libX11-1.7.2.tar.bz2.zip

libX11-1.7.2-scan-cyclonedx.json.zip

libX11-1.7.2-scan-json-pp.json.zip

pombredanne commented 2 years ago

@DennisClark Thanks... that's super useful to track this issue.

AyanSinhaMahapatra commented 2 years ago

@DennisClark The above PR addresses the crash.

  1. We do not crash anymore
  2. We show the following CLI warning:

CycloneDxPluginNoPackagesWarning: The --cyclonedx-xml option will not output any component/dependency data as there are no package data in the present scan. To get package data please rerun the scan with --package or --system-package CLI options enabled.

  1. We also add the same message to the BOM metadata in warnings.

Please review the warning text also btw.

As for the did not produce really interesting results and produces a very limited file with hardly any content part of the issue, we need some more work and investigation. So let's keep this issue open.

DennisClark commented 2 years ago

@AyanSinhaMahapatra The warning text provided in your comment is good and very informative, thanks. No changes suggested.

DennisClark commented 2 years ago

a helpful link here: https://cyclonedx.org/schema/bom-1.4.schema.json

DennisClark commented 2 years ago

another helpful link: https://cyclonedx.org/use-cases/

pombredanne commented 2 years ago

The bug is fixed and we are keeping this open to ensure we can report more useful data.

AyanSinhaMahapatra commented 1 year ago

From @pombredanne at this comment:

please add an issue for follow up work to:

  • support newer/newest versions of the spec
  • add support for dependencies and more details
AyanSinhaMahapatra commented 1 year ago

See a sample output at https://github.com/nexB/scancode-toolkit/issues/3016#issuecomment-1397144996 for reference

eastmadc commented 1 year ago

CycloneDX v1.5 was released last month. @pombredanne do you think it would be worthwhile to add this support? Would you maintain legacy version support as well? https://cyclonedx.org/news/cyclonedx-v1.5-released/

pombredanne commented 1 year ago

@eastmadc sure! I think the work will consist in migrating to this library https://gitlab.com/hoppr/hoppr-cyclonedx-models/-/tree/main/hoppr_cyclonedx_models that we already use in ScanCode.io and already has support for CycloneDX 1.5 (@jhlmco and team Thank you BTW.. ! )

Would you maintain legacy version support as well?

I think we should.