Stupid large file sizes for pickled data

seallard / walker

NEAT

MIT License

0 stars 0 forks source link

Stupid large file sizes for pickled data #79

Closed seallard closed 3 years ago

seallard commented 3 years ago

30 runs resulted in 11 Gb pickled data. That is stupid and takes forever to write/load.

seallard commented 3 years ago

Alternatives to pickle? If pickle, use higher protocol for dumping. Also reduce the number of objects, only store actual data of interest.

seallard commented 3 years ago

https://www.benfrederickson.com/dont-pickle-your-data/

seallard commented 3 years ago

While it is nice to have the actual genomes saved to be able to rerun specific evaluations, it is not really necessary. Use JSON and just store the data of interest.