The ESP32 Data Logger is a cost-effective data acquisition system designed to support user-defined sensor interfaces. Users can bring their own sensor library and add callbacks to this repository, enhancing its functionality and adaptability for various data logging needs.
To serve this large dataset efficiently to a web browser for plotting, you can employ several strategies to reduce the amount of data transferred and displayed.
Downsampling
Downsampling involves selecting a subset of the data to represent the entire dataset. You can skip rows at a regular interval to reduce the number of data points sent to the browser. For example, you could select every 10th or 100th row, depending on the size of the dataset and the level of detail required.
Strategy:
Maintain a copy of collection config, with additional features:
Last 15 min sampled time
Last 1 hour sampled time
Last day sampled time
Aggregation
Instead of sending raw data points, you can aggregate the data over specific time intervals (e.g., hourly, daily, weekly). You can calculate the average, minimum, maximum, or other statistical measures for each interval and send these aggregated values.
On-Demand Data Loading
Instead of sending all the data at once, implement a system where the browser requests data in chunks as needed. This is similar to pagination in web applications. The browser can request data for a specific time range or a specific number of data points.
Implementation Example: Downsampling
Let's consider downsampling as a straightforward approach. You can implement this in your ESP32 code to serve the data efficiently.
qiweimao
commented
3 months ago
To serve this large dataset efficiently to a web browser for plotting, you can employ several strategies to reduce the amount of data transferred and displayed.
Strategy: Maintain a copy of collection config, with additional features:
Aggregation Instead of sending raw data points, you can aggregate the data over specific time intervals (e.g., hourly, daily, weekly). You can calculate the average, minimum, maximum, or other statistical measures for each interval and send these aggregated values.
On-Demand Data Loading Instead of sending all the data at once, implement a system where the browser requests data in chunks as needed. This is similar to pagination in web applications. The browser can request data for a specific time range or a specific number of data points.
Implementation Example: Downsampling Let's consider downsampling as a straightforward approach. You can implement this in your ESP32 code to serve the data efficiently.