search

EllieKallmier / ppa_analysis

Updates and additions to Nick Gorman's MSAT-PPA tool for 24/7 PPA project
MIT License
0 stars 2 forks source link

cache sizes #4

Open nick-gorman opened 1 month ago

nick-gorman commented 1 month ago

Currently the cache size come to about 25 GB (back_flow_notebook.ipynb). I think they could be reduced in size in two ways:

  1. use the same cache for nemosis and nemede as at the moment I think they are doubling up on some data files. I think this will about half the total cache size.
  2. switch ppa_analysis and nemed the setting file_format="parquet" and keep_csv=False. I think this might then cut the file size by 90% or so.

This would be a nice usablity improvment as most users are probably not expecting a PPA analysis tool to soak up 25 GB of disk space. In fact neither was I and accidentally used up all the remaining disk space on my computer when running the back_flow_notebook.ipynb.

EllieKallmier commented 1 month ago

Very happy for these changes to be implemented as well!