Spine DB stores parameter_values as JSON blobs in the database. This is fine as long as the blobs are small. When blobs are large objects like a time series, or a shape file. Loading the value can have significant delays, and this freezes the UI (or at least the user can't see something is happening in the background) while the blob is loaded and parsed.
There are several expensive step in this process:
Retrieve the blob from the DB (we don't know beforehand the size of the blob). This is an uninterruptible asynchronous step
Accessing the DB requires disk I/O or connecting over the network (both are potentially slow)
Parsing and allocating memory to store the object (either as Python/Arrow objects)
Spine DB API overhead in creating the Timeseries data structure
Documentation
ESI-FAR/wiki#61
Different cases that should be benchmarked
Create example scripts that replicate and benchmark different options:
Spine DB stores
parameter_values as JSON blobs in the database. This is fine as long as the blobs are small. When blobs are large objects like a time series, or a shape file. Loading the value can have significant delays, and this freezes the UI (or at least the user can't see something is happening in the background) while the blob is loaded and parsed.There are several expensive step in this process:
Timeseriesdata structureDocumentation
Different cases that should be benchmarked
Create example scripts that replicate and benchmark different options:
binary_cols.py~What to benchmark?
Possible data types
parameter_values can be of various types. (the first two are not relevant here, included for completeness)Example datasets
Spine data structure
https://github.com/energy-modelling-workbench/spine-data-model#spine-data-model
For more notes on when/how these are used, see ESI-FAR/wiki#44 (the visualisation issue)