search

calliope-project / euro-calliope

A workflow to build models of the European electricity system for Calliope.
https://euro-calliope.readthedocs.io
MIT License
31 stars 18 forks source link

Always compress CSVs #206

Open brynpickering opened 2 years ago

brynpickering commented 2 years ago

What can be improved?

Timeseries output can be very large, especially with the addition of multiple years of data (#182). The easiest way to reduce the footprint of the model on disk is to compress the CSVs by default. This can reduce CSV size by 80-90% with minimal additional read/write time.

Pandas can infer this on both read and write. I.e., if a file to be saved ends in .zip, then pandas will know to compress it. Likewise if it reads a file that ends in .zip.

Version

1.2.0.dev

brynpickering commented 2 years ago

@timtroendle: would you recommend that at least the intermediate .csv workflow files should now be .feather files?

timtroendle commented 2 years ago

I would wait a little to gather more experience, but in general: yes, I think we should do that.