search

davidclemens / Dingi

Dingi is a toolbox for processing data collected by marine research gear.
MIT License
1 stars 0 forks source link

`dataPool`: Improve performance #117

Open davidclemens opened 1 year ago

davidclemens commented 1 year ago

Current Implementation

The current fetchData method is really slow.

Suggested Enhancement

Improve the performance. Possible aspects to look into:

  1. Change the storage backend: Instead of cells use vectorized storage with a lookup table.
  2. Change the aggregation/return method: Make the returned data structure much more simple.
  3. Seperate the data storage and metadata storage into a super- and subclass: e.g. DataKit.dataStore as superclass and dataPool < DataKit.dataStore.

Expected Benefits

Improved performance.

Is this something you're interested in working on

Yes.