search

davidclemens / Dingi

Dingi is a toolbox for processing data collected by marine research gear.
MIT License
1 stars 0 forks source link

`bigoFluxAnalysis`: Improve raw data storage backend #118

Open davidclemens opened 1 year ago

davidclemens commented 1 year ago

Current Implementation

The raw data is stored in an array that has as many rows as the longest dataset. Data sets with less data are padded with NaNs.

Suggested Enhancement

There are some solutions:

  1. Store as a sparse array
  2. Store in a vectorized format, e.g. as DataKit.dataStore (see #117)

Expected Benefits

Smaller storage footprint.

Is this something you're interested in working on

Yes.