Latency between Unity marker stream recorded in Lab Streaming Layer and Unity marker stream recorded in Emotiv

[gomsing]

I am currently building a Unity based experiment with Emotiv Flex, saline.

I set up a simple oddball task with 1 sec display of an object and a isi of 500 ms.

I recorded the marker stream sent from Unity both on LabRecorder (LSL) and received the marker stream on EmotivPro for visualization purpose.

However, when I checked the data, there is a 60ms to whooping 200ms latency between Unity marker stream (LSL) and Marker stream recorded with the EEG data. There are inconsistent jitters also, which makes it almost impossible to correct them in the preprocessing stage.

This is a serious issue as EEG experiments are very time sensitive and 200ms latency ruins the sanity of the data.

Which marker timestamp is correct? or are they both wrong? I have included the marker plot for illustration.

In conclusion, the timestamp values on the Unity marker stream is different depending on which program (LSL, EmotivPro) records them.

PS. There are also loss of Markers in EmotivPro, too (see the marker between 6 and 7, which only exists on Unity marker)

[hoangphamemotiv]

Hi @gomsing

Thank you for your valuable feedback. EEG Timestamp and Inject markers are one of our key features, and we handle them with utmost care.

Regarding the reported issue, there are a few aspects to consider:

1. Marker Transmission Delay: Check the time delay when a marker is sent from Unity to EmotivPro.

   • Consider sending the marker time along with the marker value to ensure it is unaffected by the transmission method.
   • Explore our API https://emotiv.gitbook.io/cortex-api/headset/syncwithheadsetclock for syncing with the headset clock. This API offers a high-performance solution, leveraging steady clock time to overcome the instability of computer lock time. (I will confirm with the development team whether this API is supported in the Unity Plugin.)

2. Marker Timestamp Method:

   • Examine how the marker timestamps are set.
   • Marker Creation Performance

[gomsing]

Hello @hoangphamemotiv,

Thank you for your prompt response.

The marker time was recorded on the LSL, along with the marker value.

In Unity, a marker is sent during trial onset and offset.

But the LSL time (timestamp) value does not match among two methods, which is weird considering they were recorded at the same time on the same LSL clock.

I merged the two dataframes along the 'timestamp' with the latency value (dif) to clarify.

I will try to record the time on Unity and compare it with the LSL time and Emotiv time.

And I was wondering whether there is a set delay of Emotiv data recording?

If both the marker and the data was delayed I can just correct it, but I am worried that only the marker was delayed whereas the EEG data was real time.

[hoangphamemotiv]

@gomsing .

• The EEG timestamp processing is independent of other processes; it is not delayed by any application process.
• A marker is incorporated into an EEG stream based on its timestamp.
• If a marker's timestamp is added before being sent via LSL, there is no impact from the transmission layer thereafter.
• Therefore, we should ensure that the timestamp is added to the marker as soon as possible after its created.

[gomsing]

@hoangphamemotiv

Thank you for a prompt response.

I think I may have phased the question in a confusing way, so let me clarify and start over.

So this is my setup, the markers are sent from Unity via LSL4Unity. The marker stream was recorded both on LabRecorder (see bottom left) and via LSL Emotiv Inlet. We streamed the data via inlet only for visualizing purpose, but checked it anyway to see whether we can just use one data stream instead of two.

However, when I use the LSL timestamp to align the two data, there is a mismatch, with markers recored via Emotiv Inlet being 60~100ms behind.

So based on this and your response I have a few questions.

When you say "a marker's timestamp is added before being sent via LSL" does this mean I should include the timestamp value from Unity in the marker stream? will it be incorporated via that value (and what clock is the Emotiv timestamp based on?)

Also, if that is the case, why is there as delay when it is incorporated into Emotiv when it was based on LSL timestamp value? Could the latency due to the process of Unity sending the marker stream to LSL -> LSL sends the received Unity marker stream to Emotiv?

Moreover, I noticed the 'transition time' in the inlet stream page. Could you please give me more information on this? although 0.118 ms should hardly affect anything

Thank you for your time.

[hoangphamemotiv]

@gomsing "Does this mean I should include the timestamp value from Unity in the marker stream?" - Yes, that's correct.

From your image, I see that the "number of channels" is set to 1, which means it only includes the marker value and doesn't have the marker timestamp.

It should be set to 3. You can refer to this link: (https://github.com/Emotiv/labstreaminglayer/tree/master/examples/unity#how-to-send-marker-from-unity-to-emotiv-lsl-inlet ) to include the timestamp of the marker and the time point when it's sent via LSL.

The timestamp should be in Epoch Time format, (https://www.epochconverter.com/). Then, you can use it for markers.

The "transition time" represents the transmission time from when the marker is sent via LSL to when it's received by EmotivPro. I noticed that it's very small at 0.1 ms, so the issue likely doesn't originate from there.

[gomsing]

@hoangphamemotiv

AH! thank you so much

I thought the LSL will take care of the timestamp, but apparently I was wrong.

Thank you for a prompt response :)

[hoangphamemotiv]

@gomsing With "number of channels" set to 1 (only marker value), EmotivPro will assign a Timestamp to the marker as soon as it's received. It will then subtract the transmission time. However, EmotivPro doesn't have information about when the marker was created. That's why Option 2, which involves adding a timestamp to the marker stream, is very useful.

I hope this clarifies the process.

[gomsing]

@hoangphamemotiv

I tried it again with the Epoch time in the Marker stream. My marker sender code is the following.

` using System.Collections; using System.Collections.Generic; using UnityEngine; using LSL; using System;

public class MarkerSender : MonoBehaviour { private LSL.StreamInfo markerStreamInfo; private LSL.StreamOutlet markerOutlet;

public int lslChannelCount = 3;

private const LSL.channel_format_t lslChannelFormat = LSL.channel_format_t.cf_double64;

private double[] sample;

public long epochNow; 

void Start()
{

    // Create a marker stream info
    markerStreamInfo = new LSL.StreamInfo("ExpMarkerStream", "Markers", lslChannelCount, 0, 
                                            lslChannelFormat, "UnityLSL001");

    LSL.XMLElement chns = markerStreamInfo.desc().append_child("channels");
        string[] channels = {"MarkerTime", "MarkerValue", "CurrentTime"};
        foreach(string chanName in channels) {
            chns.append_child("channel").append_child_value("label", chanName).append_child_value("type", "Marker");
        }

    // Create a marker stream outlet
    markerOutlet = new LSL.StreamOutlet(markerStreamInfo);
    Debug.Log("Marker stream created.");
}

public void SendMarker(int markerNum, long epochNow)
{
    sample = new double[lslChannelCount];

    if (markerOutlet != null)
    {
        // Send the marker using the outlet

        epochNow = (long)(DateTime.Now - new DateTime(1970, 1, 1)).TotalSeconds;
        sample[0] = epochNow;
        sample[1] = markerNum;
        sample[2] = epochNow;
        markerOutlet.push_sample(sample);
        Debug.Log(string.Join(", ", sample));
    }
    else
    {
        Debug.LogWarning("Marker outlet is not initialized.");
    }
}

}

`

However, the marker value received from Emotiv Inlet and received from LSL via Unity Outlet still has latency.

Now it is about 200 ms, which is not good for any EEG research.

Can you help me with why this is happening?

[hoangphamemotiv]

@gomsing can you store timestamp you added for marker in : public void SendMarker(int markerNum, long epochNow) ? then check if the marker add into correct EEG with closest timestamp. so the delay = Marker_Timestamp - EEG_Timestamp which added marker.

[gomsing]

@hoangphamemotiv

So I merged the marker stream data and eeg data based on LSL timestamp, and then compared the recorded Epoch time.

MarkerValue is the marker from the Unity LSL marker stream, and the Markers are the inlet stream.

The marker recorded on inlet has some data losses, and it has a consistent latency of 32400 epoch time.

[hoangphamemotiv]

@gomsing I don't be clear your test yet, maybe we are talking with different terms. what are "LSL timestamp", "the recorded Epoch time"? How did you draw your table? i see MarkerTime some row is NaN? and it doesn't show timestamp exactly to ms. (it is e+09). so it is hard for me to check. can you send me some delay values from this:

the delay = Marker_Timestamp - EEG_Timestamp which added marker.

[hoangphamemotiv]

and i see you are using: epochNow = (long)(DateTime.Now - new DateTime(1970, 1, 1)).TotalSeconds; seem marker timestamp is second. but it should be miliseconds

[gomsing]

@hoangphamemotiv

So what I mean by LSL timestamp and recorded Epochtime is that I merged the two data streams (marker and eeg) according to the LabStreamingLayer clock(LSL timestamp) and compared the recorded epochtime between the markers recorded in unity and markers recorded (received) in emotiv.

If the marker was aligned correctly, the epochtime should have the same values as it was aligned with the EEG data within the EmotivPro.

The delay value is the 'dif' in the data table, and the NaN value is there because the marker timing does not match (which is the problem here)

I will try again with the miliseconds, but in the reference you gave me, it used TotalSeconds.

https://github.com/Emotiv/labstreaminglayer/blob/master/examples/unity/Assets/LSL4Unity/Scripts/LSLMarkerStream.cs

[gomsing]

I changed the epochNow to miliseconds, and the difference between the timestamp in EmotivPro and the timestamp recorded in Unity is too big, the EmotivPro seems to run on total seconds?

FYI I changed the 'Timestamp' name to EEGtime~ to clarify that it was the timestamp of the inlet markers in EMotivPro

[hoangphamemotiv]

Timestamp in Emotivpro is seconds. (exact to ms) marker timestamp from LSL is miliseconds . => to compare time. it need to be: /1000 to convert to seconds

[gomsing]

@hoangphamemotiv

These are the recorded delay values

[hoangphamemotiv]

Can you send me your recording file exported from EmotivPro and MarkerTime saved from Unity outlet? so, we have 2 files.

[gomsing]

Both are recorded through LabRecorder (https://github.com/labstreaminglayer/App-LabRecorder)

So the 'timestamp' is the LSL timestamp, and "Timestamp" is the emotiv/unity epochtime.

Here are the two recording files. EEG.csv Marker.csv

[hoangphamemotiv]

is that correct files? i see timestamp much different.

[gomsing]

The left is the raw EEG data with the markers, and the right is the Marker data with only the event marker values and the timestamp. The data with only the markers is here. There are many marker drops (as you can see on the plot), so the data size may not match. markerEEG.csv

[hoangphamemotiv]

@gomsing . so sorry, but i haven't had what i need for the checking. your marker file had timestamp very different with EEG timesamp. can you explain about that?

[hoangphamemotiv]

Our QA tested the feature again. And it works fine.

[gomsing]

@hoangphamemotiv

I will look into the timestamp, I did not notice that.

The feature worked fine for us, too, but the marker received from inlet was just recorded at a different time from the data recorded from the Unity outlet.

+) The Emotiv Pro doc (https://emotiv.gitbook.io/emotivpro-v3/lab-streaming-layer-lsl/lsl-inlet) says that the time format should be in double format. Which is correct?

[hoangphamemotiv]

Thanks @gomsing and waiting for the marker file with correct timestamps. I'm trying to check step by step on your flow. the first step is ensure a marker is added correct into EEG stream with correct timestamp. Yes, timestamp is double to store a big number of Epoch time.

[gomsing]

@hoangphamemotiv

I checked my original files, and it seems that the all the timestamps are recorded wrong, which caused the latency (I think). I will check my codes, try again, and get back to you. Thank you.

[gomsing]

@hoangphamemotiv Whatever I try, the Epochtime recorded in Unity and the Epochtime recorded in Emotiv seems to be off.

That is why it wont align correctly in EmotivPro. The attatched files are the most recently recorded ones, but they still have vastly different timestamps although they are recorded at the same time.

But I'm lost why this would happen?? I've attatched my unity scripts that I use to send the markers to LSL.

Marker.csv EEG.csv Scripts.zip

[gomsing]

@hoangphamemotiv

So I checked the Epoch timestamps, and the Emotiv timestamps are not the current time (about 9 hours earlier than local KST time) whileas the Epochtime stamps in Unity is based on current(KST) time.

In short, the timestamp values streamed from EmotivPro to Lab Streaming Layer is not recording the local (kst) time.

[hoangphamemotiv]

That is what i'm thinking. Then, i did: Emotiv_Timestamps -32400 (s) (9 hours), then compare with marker time. here's the result below

Delay.xlsx

Delay time has maximum ~4ms = 1/2 sample_time as we expected. 1 sample_time = 1/ sample rate = 1/128Hz ~= 3.9ms. So EmotivPro worked fine.

about Timezone. i trust EmotivPro. if you send me original recording file exported from EmotivPro, it will show all related information. (your EEG file was modified)

Note:

• EEG timestamp is: time point of an EEG was sampled in Emotiv headset. -> when you received it from software, applications, it is the past. (it need time to run from Emotiv Headset -> Cortex -> application/data processing and can be delayed by manything. but note that, timestamp is not affected by transmission time.) -> you should not calculate the delay based on time when you received EEG. So, i'm not sure the way you are using LabRecorder to check timestamp is correct.

[gomsing]

@hoangphamemotiv P011_EPOCFLEX_201215_2023.11.14T15.10.00+09.00.md.bp.csv

P011_EPOCFLEX_201215_2023.11.14T15.10.00+09.00_intervalMarker.csv

Here are the files exported from Emotiv Pro, the epochtime is the same as the one recorded in LabRecorder, 9 hours earlier than local time

[hoangphamemotiv]

Thanks @gomsing . so from the file name, we can see it handled the your timezone +09. and it recorded : 15.10.00 today. so, i guess, your time now is about 3 PM.

There are some suggestions:

• As i noted above, when you created a marker from software/apps with timestamp = A. it will be added to an EEG sample EXACTLY just sampled in Emotiv headset, not add to a sample run to software that time point. -> we need to wait x (ms) more to received the correct EEG sample. -> you will get correct EEG sample at: A + x. BUT with timestamp = A. x can be affected and changed by manything. but A never changed.
• If you want to add markers to EEG stream yourself, you should store Marker timestamp, and find the closest timestamp from an EEG sample manually to add.

[gomsing]

@hoangphamemotiv

One question, why aren't the timezone reflected in the timestamps when the system clearly receives it?? This is a big problem as the alignment, as you said, is done via Epoch time. This clearly defeats the purpose of sending Epochtime in the marker stream for proper marker alignment in EmotivPro.

I guess I can just use the LSL time and the marker timestamps recorded with Unity LSL WITHOUT sending it to EmotivPro, if the EEG samples have little to no latency of sending the data to LSL (as you said before).

If I want to add the markers to EEG stream, should I add +9 hours to my Unity Epochtime? because the closest timestamp is currently non-existnet.

[hoangphamemotiv]

@gomsing i think EEG Timestamp is: epoch time with time zone +9. Timestamp went with Markers and send to EmotivPro via LSL also Epoch time with timezone. because, if marker time and EEG time was not close, EmotivPro couldn't map them.

I'm not sure about timesamp in your Marker.csv file.

[gomsing]

@hoangphamemotiv

Ahhhhhhh, I now see why this happended.

The EEG timestamps are epoch time with GMT +9, and the Markertime recorded KST+9.

But this is weird because I did not touch anything in the Unity code and just used

epochNow = (double)(DateTime.Now - new DateTime(1970, 1, 1)).TotalMilliseconds / 1000;

To record the marker timestamps. maybe I shouldnt use DateTime.Now?

[gomsing]

@hoangphamemotiv

I changed it to epochNow = (double)(DateTime.UtcNow - new DateTime(1970, 1, 1)).TotalMilliseconds / 1000;, and now I gives me the expected values.

Still have the issue of latency in the LSL recorded data, but I think that is not the problem regarding Emotiv.