search

NREL-Sienna / InfrastructureSystems.jl

Utility package for Sienna's simulation infrastructure
https://nrel-sienna.github.io/InfrastructureSystems.jl/
BSD 3-Clause "New" or "Revised" License
35 stars 20 forks source link

Allow bulk writing and reading of time series #341

Closed daniel-thom closed 3 months ago

daniel-thom commented 3 months ago

This is at least a partial fix of https://github.com/NREL-Sienna/PowerSystems.jl/issues/1076. The current behavior is to open and close the file once per addition of a time series array. The open/close has non-trivial overhead.

This PR allows the user to open the store for bulk writes and reads.

There may be other performance issues experienced by the submitter. I’d like to see what happens with this change by itself.

This PR is targeted at main. We could easily backport it to the current IS/PSY tags.

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 98.41270% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 80.09%. Comparing base (156469f) to head (767197d). Report is 2 commits behind head on main.

:exclamation: Current head 767197d differs from pull request most recent head bfb78e3. Consider uploading reports for the commit bfb78e3 to get more accurate results

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/NREL-Sienna/InfrastructureSystems.jl/pull/341/graphs/tree.svg?width=650&height=150&src=pr&token=I73yjxYxgn&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NREL-Sienna)](https://app.codecov.io/gh/NREL-Sienna/InfrastructureSystems.jl/pull/341?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NREL-Sienna) ```diff @@ Coverage Diff @@ ## main #341 +/- ## ========================================== + Coverage 79.83% 80.09% +0.25% ========================================== Files 54 54 Lines 4330 4386 +56 ========================================== + Hits 3457 3513 +56 Misses 873 873 ``` | [Flag](https://app.codecov.io/gh/NREL-Sienna/InfrastructureSystems.jl/pull/341/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NREL-Sienna) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/NREL-Sienna/InfrastructureSystems.jl/pull/341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NREL-Sienna) | `80.09% <98.41%> (+0.25%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NREL-Sienna#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/NREL-Sienna/InfrastructureSystems.jl/pull/341?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NREL-Sienna) | Coverage Δ | | |---|---|---| | [src/in\_memory\_time\_series\_storage.jl](https://app.codecov.io/gh/NREL-Sienna/InfrastructureSystems.jl/pull/341?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NREL-Sienna#diff-c3JjL2luX21lbW9yeV90aW1lX3Nlcmllc19zdG9yYWdlLmps) | `73.64% <100.00%> (+0.41%)` | :arrow_up: | | [src/system\_data.jl](https://app.codecov.io/gh/NREL-Sienna/InfrastructureSystems.jl/pull/341?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NREL-Sienna#diff-c3JjL3N5c3RlbV9kYXRhLmps) | `91.03% <100.00%> (+0.06%)` | :arrow_up: | | [src/hdf5\_time\_series\_storage.jl](https://app.codecov.io/gh/NREL-Sienna/InfrastructureSystems.jl/pull/341?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NREL-Sienna#diff-c3JjL2hkZjVfdGltZV9zZXJpZXNfc3RvcmFnZS5qbA==) | `92.03% <98.33%> (+0.95%)` | :arrow_up: |