Horizontal Lag in time-series Plots

[stellarpower]

The original issue I opened seems to have disappeared, I can't add details right now. But there are problems with the time sync for plots, both in signal view and Raw EEG: Will edit and fill out later; not sure why the original bug did not submit as I spent half an hour filling that out.

[cyberjunk]

Hello @stellarpower,

thanks for your feedback.

Missing plot parts on the left are typically caused by the underlying channel buffer not being large enough to hold that much history of the according channel as configured in the UI. We recently fixed at at least one such case in the Raw EEG View.

Missing plot parts on the right (=drift) are typically caused by not receiving the expected amount of samples as specified by the sample-rate of the device or other input node (e.g. due to packet loss or incorrectly configured sample-rate). You can investigate this further using the Info section of the Devices widget. For instance, UDP network streaming based solutions that use broadcasting are known to cause packet loss if too many other Wifi clients are connected.

[stellarpower]

Thanks - I haven't had any missing parts on the left so far, only lag on the right. As for raw devices, I'm usually reading from Brainflow (FreeEEG32 and serial port currently - so this should be low-latency and unlikely to suffer packet loss) or the test system, using OSC with a Muse band only for debugging. I will check the devices view for more info to confirm if anything is being lost - as I started out using the Muse, there wasn't anything to see in here, and I didn't know it showed info in the same way for Brainflow devices, so I haven't explored this widget in great detail.

I'd initially wondered if the gaps in the plots were showing latency, but then after some severe drift, figured it was likely a bug. In particular, when I took this screenshot below, there's about a two second displayed lag between the onset of an alpha burst from the test system, and the reward score (seen in the white graph at the top):

I use gamma ramps on my display to provide the feedback, so from memory, I think it looked as though the screen changed brightness began pretty soon after I saw the onset of the alpha activity - but the plot of the reward score was still pretty far behind. So I'm not necessarily sure if it's an input rate problem, and without knowing that the plots display the latency, a drawing issue seemed the most likely cause. Do you think that's possible, or do oyu think this would be indicative of latency making its way through the engine?

Thanks!

[stellarpower]

I think I should change this bug (or make another) to consider some possible UI improvements so that it is clearer from the signal view when the engine may be lagging. It wasn't obvious to me that plot issues might largely be latency problems - as an example, the window was blank just now. I had to zoom really far out and go to the settings to enable latency display to confirm, so I think a simpleish fix that e.g. highlights around the pot in red when latency issues are detected would be good.

However, is there potentially another issue here? I'm just reading in from OSC at 250Hz and visualising, and there's apparently a latency of one minute:

This is on a 32-core server and Neuromore is using ~1100% CPU. Obviously such a simple protocol shouldn't be exhausting the machine. If I cut the datastream so that no OSC packets are received it's even worse, about 15 cores' utilisation. I don't know on the internal design but I'd guess sections should spin down when no data is received - this is without the session running either, just in "design mode", and with no state machine. So it would suggest the possibility of a loop stuck busy waiting or something similar.

I have tried a couple of different builds, I will need to look into it to see when it reproduces or not.

[stellarpower]

Related: #304

[stellarpower]

I've also noticed that when I deliberately set my OSC node with a lower sample rate than the stream, I get the latency indicator bars drifting to the left - even though I have data appearing on the far right of the plot, as we produce it at the source faster than we consume it in the Studio. What exactly are the latency indicators showing? I'm on a 64-core machine right now with Neuromore spinning up about 15 of them, just taking the OSC data and plotting it - and within a minute the latency displayed is more than 5 seconds. So I think something has to be up here.

When training previously using a much lower-powered system and using a Muse band, there wasn't this much of a delay between say an eyeblink and the artefact in the signal view, and I can also see it on the MuseMonitor app forwarding via OSC. So the latency indicators may be showing false positives, or perhaps something is busy waiting cause this is an enormous amount of CPU to be using just to read data in and graph it.

[stellarpower]

Is it definitely not possible this is a plotting issue? I just tapped my electrodes, and the full classifier seems to update immediately, as expected. My screen turned bright red (not visible in the screenshot) as the tapping was detected as an artefact, so that means that data is going all the way from capture through to OSC output in realtime, even though the plots all seem to indicate a lag.

[stellarpower]

Okay, there's definitely some sort of bug going on here, as this situation below is impossible:

I generally have a certain amount of lag like this, but in this particular example we can now see nodes further down the chain plotting features earlier in time than the nodes that feed into them. The reward score can't go up before the reward bandpower has responded, and the signal okay node is the logical intersection of the two conditions for artefacts at the bottom, so also makes no sense for it to become zero before the other two. So I think either timestamps are being miscalculated somewhere, or the plotting widget is not correctly drawing those timestamps.

If the view output node has a ChannelReader to get its data, is it possible that there can be mismatched latency between those?