Currently, while in the memory dataframe mode, the dataframes are kept in ram, which is intended, but also being written (AND overwritten!!) on every save of the run. This leads to hundreds of MBs being unnecessarily written, causing a lot of disk IO.
We intend to implement a check whether a file really needs to be (over)written and also a way to clear all unnecessary files, that no longer are of use in the current run.
Acceptance Criteria
[ ] Add a checker whether a file really needs to be written to disk. (or something equivalent)
[ ] Files are no longer being overwritten if the contents did not change
[ ] Outdated dataframes are deleted, avoiding a cluttering of the run directory.
Issue description
Currently, while in the memory dataframe mode, the dataframes are kept in ram, which is intended, but also being written (AND overwritten!!) on every save of the run. This leads to hundreds of MBs being unnecessarily written, causing a lot of disk IO.
We intend to implement a check whether a file really needs to be (over)written and also a way to clear all unnecessary files, that no longer are of use in the current run.
Acceptance Criteria