Realtime plot lagging behind

[g-p-g]

While the library plots everything it receives, that doesn't seem a good idea for real time plots. The easiest way to visualize that is by using the well known method of switching tabs. If during that time lots of new updates are received, the plot never catches up with the data that's actually new. So, is there some simple way to discard irrelevant data for realtime plots?

I'm using history size of 100, and queue size of 50 (if those matters for the issue).

[g-p-g]

Using a much shorter queueSize and historySize (1 for each) seems to almost solve the problem. The remaining issue is that even when you're in the tab the plots lag behind significantly (where in this case significantly is ~0.5 second) -- at least on my computer.

[rsandor]

@g-p-g - That is a very valid point, and definitely something worth considering soon.

So the original reason that we have a queue between the incoming data and the actually rendered points was to deal with variable latency from the source. Imagine, if you will, getting a burst of multiple points and then lagging for a couple of seconds. This is a scenario that we ran across when I was building out the original Fastly analytics dashboard. Putting a queue between the source and the incoming plot data helps "smooth" out the transitions and gives the plot a much better look and feel (with the added benefit of not losing data).

This reasoning aside, your point definitely stands. For your purposes if you don't mind losing data then the solution of setting the queue and history sizes to 1 will work. But I have also been toying with the idea of adding full playback controls to plots that allow you to simply "fast forward" without having to miss large runs of information.

This sort of behavior will require a bit more structure with how you interact with graphs. My thoughts at this time are to begin by building out a Data Source class that manages and formats incoming data, fires events when new data comes in, and provides a simple way perform playback control actions across multiple charts.

As far as the 500ms latency is concerned, I wonder if the playback controls might be able to help out with that. Thoughts?

[g-p-g]

@rsandor I feel you, I agree that queueing is a good approach with bursts of data so you don't actually slow down the user's browser. The playback control is really interesting, I suppose you're going to limit how long it can play back to restrict memory usage.

If at some point such "fast forward" method comes to live, and that doesn't miss significant data, then it looks like it should be an option when starting the plot and I would be using it for data where losing some points is not that really important (in my case I can reconstruct most of the data, and several times I have to quickly interpolate to build a smooth plot). The latency is annoying but so far I'm living with it with queue size = history size = 1.

Thanks for looking into, the playback idea looks cool :)

[mattkrick]

+1 for this. For my use case, every second I get 25 new datapoints (each represents a 40ms interval). if i reduce the queue below 25, I lose data, but if it's higher than 1, the chart goes slower than the data comes in. What would be nice is a interval option. By saying something like interval: 40 we know the next point is 40ms away & since it's already delayed 1 second, you know that the graph should move 1/ windowSize% every 40ms. In doing so, we'd still be able to keep the smooth scroll which I admit is AMAZING and the x-axis would be true to scale (eg 100 px = 1 second).

[virtuald]

It's sad that this issue hasn't really been resolved, the library seems pretty awesome otherwise. I just spun up a simple line chart that updates at 10hz and the latency is really bad.

[rsandor]

@virtuald - yes, it would be nice to handle this specific case. Unfortunately we make an assumption about how the data will be fetched due to how the library was originally used at fastly.

Are you up for contributing? If so it would be great if you could look into the issue and possibly resolve it (or at the very least find a path to resolution).

Cheers!

[mgazic]

I'm also seeing large latency, several seconds. I've looked through the source code, but so far I haven't determined what causes it, or what to tweak to reduce it. Any advice? I really don't care about smooth animations, I want to minimize latency. I'd be happier if the chart stopped moving if no new data was fed to it and instantaneously jumped when new data appeared, rather than having several seconds of latency.

[boatbodger]

I too am trying to do real-time plots of movement detector output, and seeing something like ten seconds of latency even with history and queue Sizes set to 1. Data arrival rate is about 10Hz. Are there any workarounds at all for this, or am I just using the wrong tool?

[leeoniya]

if it's a line or area chart, you can try out https://github.com/leeoniya/uPlot