Previously, --stop was just smashing all input JSON files together, and --analyze was parsing that huge JSON file, and performing the analysis. The memory allocator used was a simple "bump a pointer, never free" allocator. On a decent size project (e.g. Unity editor build), that was taking 4.8sec / 5.8GB for stop, and 9.9sec / 7.3GB for analyze (so total 14.7sec, and max 7.3GB memory usage). The resulting file was 1.01GB size.
Now:
[x] do the JSON parsing during --stop step into the build events data structures (which includes string/name deduplication), and as a result just store the binary data structure file. --analyze reads that binary file and does no JSON parsing at all now.
[x] use regular built-in memory allocator (saves a ton of memory for large analyses).
[x] multi-threaded --stop part that parses all the JSON files, using enkiTS task scheduler.
[x] use xxHash hash function instead of C++ built-in one.
[x] optimize "Template sets" and "Function sets" analysis calculation
On the same Unity editor build: 2.4sec / 0.35GB for stop, and 2.7sec / 0.51GB for analyze (so total 5.1sec, and max 0.5GB memory usage). The resulting file is 280MB size.
However the data files produced by --stop and used by --analyze are no longer compatible with previous versions (previous were just JSON, now custom binary).
Redo how the data is gathered & stored to better scale for large builds (ref #10, #30).
TLDR: 3x faster, 15x lower memory usage, 4x smaller file size.
Previously,
--stopwas just smashing all input JSON files together, and--analyzewas parsing that huge JSON file, and performing the analysis. The memory allocator used was a simple "bump a pointer, never free" allocator. On a decent size project (e.g. Unity editor build), that was taking 4.8sec / 5.8GB for stop, and 9.9sec / 7.3GB for analyze (so total 14.7sec, and max 7.3GB memory usage). The resulting file was 1.01GB size.Now:
--stopstep into the build events data structures (which includes string/name deduplication), and as a result just store the binary data structure file.--analyzereads that binary file and does no JSON parsing at all now.--stoppart that parses all the JSON files, using enkiTS task scheduler.sajsonJSON parser with simdjson.On the same Unity editor build: 2.4sec / 0.35GB for stop, and 2.7sec / 0.51GB for analyze (so total 5.1sec, and max 0.5GB memory usage). The resulting file is 280MB size.
However the data files produced by
--stopand used by--analyzeare no longer compatible with previous versions (previous were just JSON, now custom binary).