Error when reading Neuroscan .cnt files using mne.io.read_raw_cnt()

arjbingly commented 3 weeks ago

Description of the problem

I'm encountering an IndexError when attempting to load Neuroscan .cnt files with mne.io.read_raw_cnt() in MNE-Python version 1.8.0. This issue does not occur in version 1.3.0, suggesting a potential regression.

Additional Observations:

When using preload=False, the error occurs upon calling raw.load_data().
When using mne.io.read_raw_cnt(file_path, preload=True) (hereby referred to as Code1), a RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.. The default option seems to be right for the file.
However, when specifying attribute data_format='int16' (hereby referred to as Code2), a ValueError: cannot reshape array of size 5330556 into shape (781250,64,1) is produced.
With Code2 and preload=False, loading small portions of the data (raw.get_data(start=0, stop=100)) does not produce an error.

Steps Taken:

The .cnt file was produced by Neuroscan.
The file structure was manually parsed using Kaitai, confirming the file's integrity (I can share the data structure and associated code I used to read the file.).
I did not encounter this issue in MNE 1.3.0, which indicates a regression, but I’m not sure which specific version introduced the issue.

Steps to reproduce

import mne
raw = mne.io.read_raw_cnt(file_path, preload=True) #Code1
raw = mne.io.read_raw_cnt(file_path, data_format='int16', preload=True) #Code2

Link to data

Unfortunately, I am not allowed to share the data.

Expected results

Execute without errors.

Actual results

The error produced with Code1

src/read_cnt_mne.py:13: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  data = mne.io.read_raw_cnt(file_path, preload=True)
src/read_cnt_mne.py:13: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  data = mne.io.read_raw_cnt(file_path, preload=True)
Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-9998220a705c>", line 1, in <module>
    runfile('src/read_cnt_mne.py', wdir='src')
  File "/Users/user/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "src/read_cnt_mne.py", line 13, in <module>
    data = mne.io.read_raw_cnt(file_path, preload=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/mne/io/cnt/cnt.py", line 248, in read_raw_cnt
    return RawCNT(
           ^^^^^^^
  File ".venv/lib/python3.12/site-packages/mne/io/cnt/cnt.py", line 531, in __init__
    info, cnt_info = _get_cnt_info(
                     ^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/mne/io/cnt/cnt.py", line 340, in _get_cnt_info
    if annotations.onset[-1] * sfreq > n_samples:
       ~~~~~~~~~~~~~~~~~^^^^
IndexError: index -1 is out of bounds for axis 0 with size 0

The error produced with Code2

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-9998220a705c>", line 1, in <module>
    runfile('src/read_cnt_mne.py', wdir='src')
  File "/Users/user/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "src/read_cnt_mne.py", line 13, in <module>
    data = mne.io.read_raw_cnt(file_path, data_format='int16', preload=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/mne/io/cnt/cnt.py", line 248, in read_raw_cnt
    return RawCNT(
           ^^^^^^^
  File ".venv/lib/python3.12/site-packages/mne/io/cnt/cnt.py", line 535, in __init__
    super().__init__(
  File "<decorator-gen-185>", line 12, in __init__
  File ".venv/lib/python3.12/site-packages/mne/io/base.py", line 301, in __init__
    self._preload_data(preload)
  File ".venv/lib/python3.12/site-packages/mne/io/base.py", line 596, in _preload_data
    self._data = self._read_segment(data_buffer=data_buffer)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<decorator-gen-187>", line 12, in _read_segment
  File ".venv/lib/python3.12/site-packages/mne/io/base.py", line 472, in _read_segment
    _ReadSegmentFileProtector(self)._read_segment_file(
  File ".venv/lib/python3.12/site-packages/mne/io/base.py", line 2598, in _read_segment_file
    return self.__raw.__class__._read_segment_file(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/mne/io/cnt/cnt.py", line 605, in _read_segment_file
    samps = samps.reshape((n_chunks, f_channels, channel_offset), order="C")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: cannot reshape array of size 5330556 into shape (781250,64,1)

Additional information

Platform             macOS-15.1-arm64-arm-64bit
Python               3.12.0 (v3.12.0:0fb18b02c8, Oct  2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)]
Executable           .venv/bin/python
CPU                  arm (12 cores)
Memory               32.0 GB
Core
├☑ mne               1.8.0 (latest release)
├☑ numpy             1.26.4 (OpenBLAS 0.3.23.dev with 12 threads)
├☑ scipy             1.14.1
└☑ matplotlib        3.9.2 (backend=module://backend_interagg)
Numerical (optional)
├☑ sklearn           1.5.2
├☑ numba             0.60.0
├☑ nibabel           5.3.2
├☑ nilearn           0.10.4
├☑ dipy              1.9.0
├☑ openmeeg          2.5.12
├☑ pandas            2.2.3
├☑ h5io              0.2.4
├☑ h5py              3.12.1
└☐ unavailable       cupy
Visualization (optional)
├☑ pyvista           0.44.1 (OpenGL 4.1 Metal - 89.3 via Apple M2 Max)
├☑ pyvistaqt         0.11.1
├☑ vtk               9.3.1
├☑ qtpy              2.4.1 (PyQt6=6.7.1)
├☑ ipympl            0.9.4
├☑ pyqtgraph         0.13.7
├☑ mne-qt-browser    0.6.3
├☑ ipywidgets        8.1.5
├☑ trame_client      3.4.0
├☑ trame_server      3.2.3
├☑ trame_vtk         2.8.11
└☑ trame_vuetify     2.7.1
Ecosystem (optional)
├☑ neo               0.13.4
├☑ eeglabio          0.0.2-4
├☑ edfio             0.4.5
├☑ mffpy             0.10.0
├☑ pybv              0.7.5
└☐ unavailable       mne-bids, mne-nirs, mne-features, mne-connectivity, mne-icalabel, mne-bids-pipeline

welcome[bot] commented 3 weeks ago

Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴

MichalBures-OG commented 1 week ago

I have noticed the same problem (IndexError: index -1 is out of bounds for axis 0 with size 0) while trying to load this dataset: https://figshare.com/articles/dataset/The_original_EEG_data_for_driver_fatigue_detection/5202739.

I tested different versions of MNE and discovered that the last version where the .cnt file loading works is 1.6.1. With 1.7.0 it breaks.

ZiyiTsang commented 2 days ago

Hi, I also meet this problem in issue #12982 and submit a PR #12986 to quick fix for this issue. Please wait for the futher review. If you are hurry, you may try to change the code in MNE source code: if annotations.onset[-1] * sfreq > n_samples to if annotations and annotations.onset[-1] * sfreq > n_samples If it also wrok on your case, you may close this issue and waiting for PR merges. Thank you.

mne-tools / mne-python