cmu-delphi / epiprocess

Tools for basic signal processing in epidemiology
https://cmu-delphi.github.io/epiprocess/
Other
13 stars 9 forks source link

Add separate epi_archive constructor based on a "iterator"/list/df of snapshots #173

Open brookslogan opened 2 years ago

brookslogan commented 2 years ago

See #172 for background. This would work around some of the caveats noted there by:

brookslogan commented 1 year ago

We can be a lot more memory-friendly than an in-memory list/df of snapshots, by only reading in one / a few snapshots at a time, and compactifying along the way (plus suggesting to save the compactified version on disk if we compactified a lot, or more complicated things regarding proposed "updating" archives that are out of scope for this initial feature).