sahib
commented
3 years ago Hey @SeeSpotRun,
One side-effect of this implementation is that the json file doesn't get written until right at the end. The previous implementation writes to file as the results come in.
That's actually a crucial property for the following reasons:
- JSON files can get very big (I saw several GB in edge cases), keeping it all in memory is not a good thing.
- Current JSON has to be written on CTRL-C, for big files that will take long.
- GUI depends on the streaming of JSON to show the results in realtime.
So, yes, I think you should go with JsonGenerator. IIRC you don't need to instance a new one on each element, but can just call json_generator_set_root() on each (or so I hope to avoid the extra allocation).
SeeSpotRun
Note this PR was branched off earlier PR https://github.com/sahib/rmlint/pull/479 to simplify possible merging.
Should address #464 and #463 and make json handling more robust in future.
One side-effect of this implementation is that the json file doesn't get written until right at the end. The previous implementation writes to file as the results come in.
To do similar buffering with json-glib would require manually writing the opening and closing
[and]to file, as well as the,'s between array elements, and using a new JsonGenerator for each element. So it can be done but it's less elegant. But if we we run into problems keeping the JsonArray in memory then we might have to go that way anyway.Thoughts @sahib ?