feat(ScottPlot): Add SQLite Data Persistence and Large Dataset Visualization Sample

[agneszitte]

This PR adds a new sample demonstrating the integration of SQLite with ScottPlot for data persistence and large dataset visualization. The key focus of this sample is on how to handle large datasets efficiently while persisting the data in a local database for long-term storage.

Key Features:

• SQLite Data Persistence: The sample stores large datasets using SQLite, allowing for handling and retrieval of data over time.

• ScottPlot Visualizations: It showcases how to visualize large datasets using ScottPlot, with the following plot types:

  • SignalPlot and SignalConst: Optimized for evenly spaced X-values, ideal for large datasets.
  • ScatterPlot (with downsampling): Suitable for non-uniform X-values, with performance optimization through downsampling.
  • Heatmap: Visualizes 2D grid data.

  The chosen plot types above prioritize performance and visualization quality, especially for platforms like WebAssembly with resource constraints. Other plot types were excluded for this sample due to potential performance issues with large datasets based on ScottPlot documentation.

Sample focus:

• Demonstrates how to combine a database-driven approach (using SQLite) with ScottPlot to create performant and scalable visualizations, especially for platforms like WebAssembly where resources can be limited.
• Provides a reference for handling and persisting large datasets with interactive visualizations.

Techniques Used:

• SQLite Data Persistence: Datasets are persisted in an SQLite database for long-term storage and retrieval.
• Lazy Loading: Data is loaded in small batches to optimize memory usage and performance.
• Asynchronous Programming: async/await ensures smooth, non-blocking UI during database and plot operations.
• Garbage Collection (GC.Collect): Explicit garbage collection is triggered after memory-intensive operations to avoid memory bloat.
• State Management: The last-used plot type is saved and restored across sessions for user convenience.
• Marshalling with DispatcherQueue: UI updates are properly marshalled to the main thread using DispatcherQueue.
• Efficient Serialization with MessagePack: Large datasets are serialized and deserialized efficiently using MessagePack, minimizing memory and performance overhead.
• Thread Safety with Locking: Thread safety is maintained by locking shared resources during data manipulation.
• Plot Auto-Scaling: Automatic axis scaling ensures the plots are correctly visualized without manual intervention.

Tested on all platforms:

Windows, Desktop, WSL2, iOS, Android, WebAssembly and macOS.

https://github.com/user-attachments/assets/e2564dd4-09df-49c8-8742-1fae7ff6824d

[kazo0]

@agneszitte I am able to get a crash to happen. At least for net8.0-desktop

If I continuously press the Add Random Data I eventually crash with this:

[agneszitte]

@agneszitte I am able to get a crash to happen. At least for net8.0-desktop

If I continuously press the Add Random Data I eventually crash with this:

Thanks a lot @kazo0 I will take a look at it!

[agneszitte]

@agneszitte I am able to get a crash to happen. At least for net8.0-desktop If I continuously press the Add Random Data I eventually crash with this:

I am not able to reproduce with the latest changes. Tested on all platforms. Especially after making sure that UI Elements are manipulated on the UI Thread properly.