Determine root cause for data load issue on Batch Plotting Page - Githubissues

USACE / instrumentation

Instrumentation project issue tracking and project planning

MIT License

4 stars 1 forks source link

Determine root cause for data load issue on Batch Plotting Page #124

Closed dennisgsmith closed 11 months ago

dennisgsmith commented 11 months ago

Slow loading times when loading batch plots with large amounts of measurement data.

dennisgsmith commented 11 months ago

This issue has been mitigated by:

Optimizing the query (added indexes to timeseries_measurement table)
Added optional downsampling (LTTB)
- The Largest Triangle Three Buckets (LTTB) algorithm is used to reduce the overall number of points to some deterministic threshold, while maintaining the shape of the plot. This allows the client to consume less data and still accurately visualize the data over large time ranges.
Improved JSON parsing UI code

If this issue persists, other strategies can be used to mitigate:

Create API endpoints that conform to Plotly's schema, so less "data massaging" is needed
Move LTTB algorithm to database to minimize the amount of data loaded into memory (currently this process happens in the API for faster testing and delivery).