Reduce size of JSON sessions as much as possible

louislva / OpenActionData

Building a diverse and clean dataset of humans using the web. Open source.

https://open-action-data.vercel.app

4 stars 1 forks source link

Open louislva opened 1 year ago

louislva commented 1 year ago

We need to be efficient with storage. The JSON is needlessly large, and can be reduced, at least 10x, but let's try get 20-50x.

Some ideas:

GZip; in my tests this will get the file to 5-10% of original size
Binary representation of data (like pickle or similar); not sure how much this would help when applied in combination with GZip
Strip JSON of useless data