ENH: Collapse BAD_ACQ_SKIP in raw.plot

[etiennedemontalivet]

Feature description

Hi, I would like to know if there is any plan to support missing data / gaps in continous raw, especially regarding:

• annotations: take into account EDGE Boundary duration (time gap) when annotating
• time_as_index: get the timestamp index of a Raw that contains time gaps

Example

Let say I have two successive recordings of 5 and 3 seconds with a 2s gap in between. I have 2 channels at 100Hz, so the recording duration is 10 seconds (in real life), and I have 8 seconds of data (here I use simple sin and cos).

import mne
import numpy as np
from datetime import datetime,timedelta,timezone

# create basic info
info = mne.create_info(
    ch_names=["Ch1", "Ch2"],
    sfreq=100)

# create fake successive recordings 0 and 1
start_datetime = datetime.now().replace(tzinfo=timezone.utc)

x0 = np.linspace(0, 5, 500)
x1 = np.linspace(7, 10, 300)

k = 2 # only for viz purpose
raw0 = mne.io.RawArray(np.array([np.cos(k*x0), np.sin(k*x0)]), info)
raw0.set_meas_date(start_datetime);
raw1 = mne.io.RawArray(np.array([np.cos(k*x1), np.sin(k*x1)]), info)
raw1.set_meas_date(start_datetime + timedelta(0,7)); # I know when 2nd recording started

# concat and plot recordings
raws = mne.concatenate_raws([raw0, raw1])
raws.plot();

Now if I have an event that occured between the 4th and the 8th second, I would like to annotate it regarding the start of the experiment like this:

raws.annotations.append(onset=4, duration=4, description="4s event")
raws.plot(); 

The current behaviour is that the 4 seconds from the onset are annotated regardless of any gap/missing data. The feature would allow to annotate data that are discontinous.

The same happens with time_as_index:

raws.time_as_index(8) # would love result: array([600])

Additional comments

This is something common that recordings have gaps between one another (see the chb-mit database). If the gap is only a few seconds long that does'nt matter, one can fill with NaN or 0 as a workaround. But sometimes gaps are few hours long.

I only used the API for a few weeks, but if you think the feature is interesting and a new user can help on that, please let me know where the changes have to be made so I can fork and test it (but I feel this could have a strong impact).

Thanks a lot !

[welcome[bot]]

Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

[larsoner]

I would propose two things to get you what you want:

1. To "insert" a gap, create a RawArray with zeros, annotated (the whole time) as BAD_ACQ_SKIP, and have it in your concatenation like mne.concatenate_raws((raw0, raw_gap, raw1)). This should plot showing all zeros from 5-7 sec. Now your resulting raw instance has the entire time span, and you can use annotations and time_as_index or whatever you want to get the gap times.
2. For long gaps, we should provide a way in raw.plot() to collapse annotation skips so that they aren't shown

Note that for (1) at least when writing to FIF, we don't bother writing all the zeros during the BAD_ACQ_SKIP, so we wouldn't end up writing hours of zeros.

[cbrnr]

@larsoner so BAD_ACQ_SKIP is special? Is this documented somewhere? I only found that raw.filter filters segments separated by EDGE or BAD_ACQ_SKIP annotations separately (https://mne.tools/stable/generated/mne.io.Raw.html#mne.io.Raw.filter).

[larsoner]

@larsoner so BAD_ACQ_SKIP is special? Is this documented somewhere?

No not really documented well -- it triggers some special code when writing raw FIF so that we don't waste a bunch of bytes on data that does not exist. If you do raw.get_data(), the data in these ranges just show up as zeros (at least when you read a FIF file that has skips) as well. I guess somewhere we should start keeping track of the special annotation types we have like this...

Also FWIW MFF also has support for ACQ_SKIP and can insert these, since those recordings can also be started and stopped.

[cbrnr]

I guess somewhere we should start keeping track of the special annotation types we have like this...

Yes! Definitely! Maybe the docstring for mne.Annotations would be the best place?

[larsoner]

Sure, the Notes section would be good

[etiennedemontalivet]

Note that for (1) at least when writing to FIF, we don't bother writing all the zeros during the BAD_ACQ_SKIP, so we wouldn't end up writing hours of zeros

The BAD_ACQ_SKIP behaviour is really good to know, thanks ! I just tested your suggestion, it works perfectly.

I let you guys see what you want to do with the issue (about the BAD_ACQ_SKIP doc and the raw.plot() behaviour)

Thanks !