search

eddie-water / SubCraft

0 stars 0 forks source link

Find large data set plotting library with real time capabiltiy #13

Open eddie-water opened 1 month ago

eddie-water commented 1 month ago

See how to incorporate this into the main software arch

https://github.com/vispy/vispy/blob/main/examples/tutorial/app/fps.py

eddie-water commented 1 month ago

Try this one https://github.com/vispy/vispy/blob/main/examples/basics/visuals/mesh.py

eddie-water commented 1 month ago

Don't know why I'm having such a hard time finding a way to plot large data sets. Keep reading posts and advice on overall just avoiding plotting that many points. I'm not opposed. Makes sense. Not all 500k points will be displayed on screen of a particular resolution. I could look into down sampling my data, but I wish I didn't have to do it myself. I wish I could maintain my large data set, and then plot it in real time, and have the plotting api handle all the sub sampling by itself.

Crazy its so hard to visualize large data sets in real time either I'm just over complicating things somehow or there is genuinely a huge barrier for plotting large data sets.

eddie-water commented 1 month ago

This directory is nuts https://github.com/vispy/vispy/tree/main/examples/demo/scene

eddie-water commented 1 month ago

This is an FFT example running at 15 fps. Not good starting point but it's a loop. Feel like the CWT will slow it down

https://github.com/Kurene/plot-spectrogram-in-realtime-by-matplotlib

eddie-water commented 1 month ago

This example recommends down sampling before plotting.

https://stackoverflow.com/questions/4976303/speed-up-matplotlib

But I have a lot of questions.

How would that work in real-time?

Would need to downsample. Where? I should downsample in the Wavelet class. Would need to compute the CWT and store it in a private variable, but pass back a downsampled array.

Downsample how much? Idk. I need to consider what that does to the representation of the data.

Does "decimating" delete every other data point? Is that okay?

Or should they be averaged before decimating?

What is aggregating? Is that averaging? Who knows

eddie-water commented 1 month ago

Interactive plot

https://github.com/telemetrydb/lenspy

eddie-water commented 1 month ago

Wish there was a plotting library with this under the hood of the API.

Since my data is logarithmic, I might have to do it myself, but I'm afraid to down sample incorrectly.

Also, what "volatility information" gets lost when you downsample? Down sampling changes the nyquist limit on the data, but is that bad if we're just plotting it?

image

eddie-water commented 1 month ago

This explains how to downsample. Notice how it says to downsample for the horizontal resolution, which is what I was thinking since we don't want to lose any frequency resolution.

How does downsampling affect the accuracy of the results. Well downsampling the horizontal axis in half would technically make us lose information on half of the frequency axis, right? Or am I tripping

https://dsp.stackexchange.com/questions/42718/down-sample-incoming-stream-of-data-to-plot-a-dynamic-graph

eddie-water commented 1 month ago

VisPy two threads one for data generation and on for plotting.

https://youtu.be/uR5j4C-_Fak?si=1cJ39BC6Fr4-htUq