Master issue: Improve ScanCode resources usage (CPU, RAM, Disk) - Githubissues

aboutcode-org / scancode-toolkit

:mag: ScanCode detects licenses, copyrights, dependencies by "scanning code" ... to discover and inventory open source and third-party packages used in your code. Sponsored by NLnet project https://nlnet.nl/project/vulnerabilitydatabase, the Google Summer of Code, Azure credits, nexB and others generous sponsors!

https://github.com/aboutcode-org/scancode-toolkit/releases/

2.08k stars 539 forks source link

Master issue: Improve ScanCode resources usage (CPU, RAM, Disk) #3755

Open pombredanne opened 5 months ago

pombredanne commented 5 months ago

It would be great to optimize resources usage. In particular, these are the likely key hot spots:

memory as used in license detection (was typically 0.8 GB now closer to 1.5GB per process)
memory when return large JSON scans (which are all re-loaded in memory assembled from the cache)
CPU when running copyright detection
CPU when running license detection

Some related issues:

pombredanne commented 5 months ago

@tdruez can you reference related ScanCode.io issues @JonoYang same on PurlDB?