NREL-Sienna / InfrastructureSystems.jl

Utility package for Sienna's simulation infrastructure
https://nrel-sienna.github.io/InfrastructureSystems.jl/
BSD 3-Clause "New" or "Revised" License
39 stars 20 forks source link

Optimize storage of time series #227

Closed daniel-thom closed 3 years ago

daniel-thom commented 3 years ago

This makes two changes:

  1. Add option to compress time series data.
  2. Change how shared time series references are stored. The earlier method caused stranded space.

These changes required the addition of a format version so that we can maintain backwards compatibility with existing datasets.

I also added some documentation. I added a description of the time series HDF5 format. I also added descriptor rules for our auto-generation code since more people are using that feature.

Regarding compression, I'm thinking that we can expose this with kwargs to the System constructor. We could provide a simple enable_compression as well as compression that take the CompressionSettings struct defined here.

codecov[bot] commented 3 years ago

Codecov Report

Merging #227 (b39a127) into master (af9e9f9) will increase coverage by 0.08%. The diff coverage is 89.83%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #227      +/-   ##
==========================================
+ Coverage   75.01%   75.10%   +0.08%     
==========================================
  Files          43       43              
  Lines        3054     3073      +19     
==========================================
+ Hits         2291     2308      +17     
- Misses        763      765       +2     
Flag Coverage Δ
unittests 75.10% <89.83%> (+0.08%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/time_series_storage.jl 77.77% <80.00%> (+2.77%) :arrow_up:
src/hdf5_time_series_storage.jl 93.37% <90.38%> (-0.61%) :arrow_down:
src/in_memory_time_series_storage.jl 71.84% <100.00%> (+0.27%) :arrow_up:
src/system_data.jl 86.26% <100.00%> (+0.05%) :arrow_up:
src/utils/lazy_dict_from_iterator.jl 96.15% <0.00%> (-0.15%) :arrow_down:
src/utils/utils.jl 61.18% <0.00%> (+0.42%) :arrow_up:
jd-lara commented 3 years ago

Regarding compression, I'm thinking that we can expose this with kwargs to the System constructor. We could provide a simple enable_compression as well as compression that take the CompressionSettings struct defined here.

Agreed