search

mne-tools / mne-python

MNE: Magnetoencephalography (MEG) and Electroencephalography (EEG) in Python
https://mne.tools
BSD 3-Clause "New" or "Revised" License
2.61k stars 1.3k forks source link

Read_raw_eyelink failure: missing headpos data in file #12516

Open scott-huberty opened 3 months ago

scott-huberty commented 3 months ago

Description of the problem

A researcher reported a problem with our eyelink reader and shared the problematic Eyelink file (ASCII).

There are 2 separate issues.

  1. This file has an empty value for recording date/time, and our reader doesn't handle that gracefully. EDIT: The researcher actually manually removed the datetime from the ASCII file, so I think it's reasonable for us to continue assuming that EyeLink will write the datetime to the file.
  2. This file says that there are head-position (x, y, distance) data, but that data isn't actually in the file, and our reader doesn't handle that either.

I'll try to open a fix this week.

Steps to reproduce

from pathlib import Path
import mne

fname = Path(".") / "to" / "problem_file.asc"
mne.io.read_raw_eyelink(fname)

Link to data

See: https://mne.discourse.group/t/problem-reading-binocular-data-with-mne-io-read-raw-eyelink/8555

Expected results

successful file read

Actual results

BUG 1:

stack trace ``` --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[2], line 1 ----> 1 raw = mne.io.read_raw_eyelink("example_data.asc") File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:62, in read_raw_eyelink(fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose) 32 """Reader for an Eyelink ``.asc`` file. 33 34 Parameters (...) 58 ``'BAD_ACQ_SKIP'``. 59 """ 60 fname = _check_fname(fname, overwrite="read", must_exist=True, name="fname") ---> 62 raw_eyelink = RawEyelink( 63 fname, 64 create_annotations=create_annotations, 65 apply_offsets=apply_offsets, 66 find_overlaps=find_overlaps, 67 overlap_threshold=overlap_threshold, 68 verbose=verbose, 69 ) 70 return raw_eyelink File :12, in __init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose) File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:107, in RawEyelink.__init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose) 104 fname = Path(fname) 106 # ======================== Parse ASCII file ========================== --> 107 eye_ch_data, info, raw_extras = _parse_eyelink_ascii( 108 fname, find_overlaps, overlap_threshold, apply_offsets 109 ) 110 # ======================== Create Raw Object ========================= 111 super().__init__( 112 info, 113 preload=eye_ch_data, (...) 116 raw_extras=[raw_extras], 117 ) File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:50, in _parse_eyelink_ascii(fname, find_overlaps, overlap_threshold, apply_offsets) 48 raw_extras.update(_parse_recording_blocks(fname)) 49 raw_extras.update(_get_metadata(raw_extras)) ---> 50 raw_extras["dt"] = _get_recording_datetime(fname) 51 _validate_data(raw_extras) 53 # ======================== Create DataFrames ======================== File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:190, in _get_recording_datetime(fname) 186 fmt = "%a %b %d %H:%M:%S %Y" 187 # Eyelink measdate timestamps are timezone naive. 188 # Force datetime to be in UTC. 189 # Even though dt is probably in local time zone. --> 190 dt_naive = datetime.strptime(dt_str, fmt) 191 return dt_naive.replace(tzinfo=tz) File ~/miniforge3/envs/mnedev/lib/python3.10/_strptime.py:568, in _strptime_datetime(cls, data_string, format) 565 def _strptime_datetime(cls, data_string, format="%a %b %d %H:%M:%S %Y"): 566 """Return a class cls instance based on the input string and the 567 format string.""" --> 568 tt, fraction, gmtoff_fraction = _strptime(data_string, format) 569 tzname, gmtoff = tt[-2:] 570 args = tt[:6] + (fraction,) File ~/miniforge3/envs/mnedev/lib/python3.10/_strptime.py:349, in _strptime(data_string, format) 347 found = format_regex.match(data_string) 348 if not found: --> 349 raise ValueError("time data %r does not match format %r" % 350 (data_string, format)) 351 if len(data_string) != found.end(): 352 raise ValueError("unconverted data remains: %s" % 353 data_string[found.end():]) ValueError: time data '' does not match format '%a %b %d %H:%M:%S %Y' ```

BUG 2:

stack trace ``` --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[2], line 1 ----> 1 raw = mne.io.read_raw_eyelink("example_data.asc") File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:62, in read_raw_eyelink(fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose) 32 """Reader for an Eyelink ``.asc`` file. 33 34 Parameters (...) 58 ``'BAD_ACQ_SKIP'``. 59 """ 60 fname = _check_fname(fname, overwrite="read", must_exist=True, name="fname") ---> 62 raw_eyelink = RawEyelink( 63 fname, 64 create_annotations=create_annotations, 65 apply_offsets=apply_offsets, 66 find_overlaps=find_overlaps, 67 overlap_threshold=overlap_threshold, 68 verbose=verbose, 69 ) 70 return raw_eyelink File :12, in __init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose) File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:107, in RawEyelink.__init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose) 104 fname = Path(fname) 106 # ======================== Parse ASCII file ========================== --> 107 eye_ch_data, info, raw_extras = _parse_eyelink_ascii( 108 fname, find_overlaps, overlap_threshold, apply_offsets 109 ) 110 # ======================== Create Raw Object ========================= 111 super().__init__( 112 info, 113 preload=eye_ch_data, (...) 116 raw_extras=[raw_extras], 117 ) File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:58, in _parse_eyelink_ascii(fname, find_overlaps, overlap_threshold, apply_offsets) 56 # add column names to dataframes and set the dtype of each column 57 col_names, ch_names = _infer_col_names(raw_extras) ---> 58 raw_extras["dfs"] = _assign_col_names(col_names, raw_extras["dfs"]) 59 raw_extras["dfs"] = _set_df_dtypes(raw_extras["dfs"]) # set dtypes for dataframes 60 # if HREF data, convert to radians File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:407, in _assign_col_names(col_names, df_dict) 405 for key, df in df_dict.items(): 406 if key in ("samples", "blinks", "fixations", "saccades"): --> 407 df.columns = col_names[key] 408 elif key == "messages": 409 cols = ["time", "offset", "event_msg"] File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/generic.py:6310, in NDFrame.__setattr__(self, name, value) 6308 try: 6309 object.__getattribute__(self, name) -> 6310 return object.__setattr__(self, name, value) 6311 except AttributeError: 6312 pass File properties.pyx:69, in pandas._libs.properties.AxisProperty.__set__() File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/generic.py:813, in NDFrame._set_axis(self, axis, labels) 808 """ 809 This is called from the cython code when we set the `index` attribute 810 directly, e.g. `series.index = [1, 2, 3]`. 811 """ 812 labels = ensure_index(labels) --> 813 self._mgr.set_axis(axis, labels) 814 self._clear_item_cache() File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/internals/managers.py:238, in BaseBlockManager.set_axis(self, axis, new_labels) 236 def set_axis(self, axis: AxisInt, new_labels: Index) -> None: 237 # Caller is responsible for ensuring we have an Index object. --> 238 self._validate_set_axis(axis, new_labels) 239 self.axes[axis] = new_labels File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/internals/base.py:98, in DataManager._validate_set_axis(self, axis, new_labels) 95 pass 97 elif new_len != old_len: ---> 98 raise ValueError( 99 f"Length mismatch: Expected axis has {old_len} elements, new " 100 f"values have {new_len} elements" 101 ) ValueError: Length mismatch: Expected axis has 8 elements, new values have 11 elements ```

Additional information

N/A

larsoner commented 3 months ago

EDIT: The researcher actually manually removed the datetime from the ASCII file, so I think it's reasonable for us to continue assuming that EyeLink will write the datetime to the file.

I think it's okay to turn this into a warning rather than an error. It's possible others have done it I suppose

scott-huberty commented 2 months ago

I haven't forgotten about this, just haven't figured out a fix for it yet (and I'm not sure there is a simple fix).

It's difficult to figure out the channel names from the Eyelink file, when the file provides incorrect information about this (as in this case, it tells us head position channels are there, but they aren't.). My best idea right now is to let the user pass in the channel names themselves.

I'll try to stop by the next office hours to discuss the issue.

larsoner commented 2 months ago

In other readers we have an exclude param

https://mne.tools/stable/generated/mne.io.read_raw_edf.html

maybe that would work?

scott-huberty commented 2 months ago

In other readers we have an exclude param

https://mne.tools/stable/generated/mne.io.read_raw_edf.html

maybe that would work?

Thanks @larsoner , I'll try this out!