search

mne-tools / mne-python

MNE: Magnetoencephalography (MEG) and Electroencephalography (EEG) in Python
https://mne.tools
BSD 3-Clause "New" or "Revised" License
2.7k stars 1.31k forks source link

info meas_date field not being saved #5908

Closed fraimondo closed 5 years ago

fraimondo commented 5 years ago

Describe the bug

Saving a big RAW object creates a split FIF file which is then not readable.

The problem arises when setting the annotations. This is the traceback:

~/dev/mne-python/mne/io/fiff/raw.py in read_raw_fif(fname, allow_maxshield, preload, verbose)
    455     """
    456     return Raw(fname=fname, allow_maxshield=allow_maxshield,
--> 457                preload=preload, verbose=verbose)

~/dev/mne-python/mne/io/fiff/raw.py in __init__(self, fname, allow_maxshield, preload, verbose)

~/dev/mne-python/mne/utils/_logging.py in verbose(function, *args, **kwargs)
     72         with use_log_level(verbose_level):
     73             return function(*args, **kwargs)
---> 74     return function(*args, **kwargs)
     75 
     76 

~/dev/mne-python/mne/io/fiff/raw.py in __init__(self, fname, allow_maxshield, preload, verbose)
    105 
    106         # combine annotations
--> 107         self.set_annotations(raws[0].annotations, False)
    108         if any([r.annotations for r in raws[1:]]):
    109             n_samples = np.sum(self._last_samps - self._first_samps + 1)

~/dev/mne-python/mne/io/base.py in set_annotations(self, annotations, emit_warning)
    763             if self.info['meas_date'] is None and \
    764                annotations.orig_time is not None:
--> 765                 raise RuntimeError('Ambiguous operation. Setting an Annotation'
    766                                    ' object with known ``orig_time`` to a raw'
    767                                    ' object which has ``meas_date`` set to'

Digging a bit more into the code, it look like the 'meas_date' field of the info is not being saved:

In [64]: orig_raw.info['meas_date']
Out[64]: (0, 2147483647)

In [65]: orig_raw.save('test2-raw.fif', overwrite=True)
Overwriting existing file.
Writing /Users/fraimondo/dev/proyectos/heart/cleaning/test2-raw.fif
Closing /Users/fraimondo/dev/proyectos/heart/cleaning/test2-raw.fif [done]

In [66]: read_raw = mne.io.read_raw_fif('test2-raw.fif')
Opening raw data file /Users/fraimondo/dev/proyectos/heart/cleaning/test2-raw.fif...
    Range : 0 ... 204800 =      0.000 ...   100.000 secs
Ready.

In [67]: read_raw.info['meas_date']

In [68]:

Steps and/or code to reproduce

Please provide a code snippet or minimal working example (MWE) to replicate your problem.

This MWE should be self-contained, which means that other MNE-Python contributors should be able to copy and paste the provided snippet and replicate the bug. If possible, use MNE-Python testing examples to reproduce the error. Otherwise, provide a small and anonymized portion of your data required to reproduce the bug.

If the code is too long, feel free to put it in a public gist and link it in the issue.

Example:

import mne

fname = mne.datasets.sample.data_path() + '/MEG/sample/sample_audvis_raw.fif'
raw = mne.io.read_raw_fif(fname)
raw = mne.concatenate_raws([raw])
raw.save('test_raw.fif', overwrite=True)
raw_read = mne.io.read_raw_fif('test_raw.fif')  # this breaks
In [68]: mne.sys_info()
Platform:      Darwin-18.2.0-x86_64-i386-64bit
Python:        3.6.5 |Intel Corporation| (default, Aug  3 2018, 14:28:41)  [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)]
Executable:    /Users/fraimondo/anaconda3/envs/nice/bin/python
CPU:           i386: 8 cores
Memory:        Unavailable (requires "psutil" package)
mne:           0.18.dev0
numpy:         1.15.1 {blas=mkl_rt, lapack=mkl_rt}
scipy:         1.1.0
matplotlib:    2.2.3 {backend=MacOSX}

sklearn:       0.19.2
nibabel:       Not found
mayavi:        Not found
cupy:          Not found
pandas:        0.23.4+0.g0409521.dirty
dipy:          Not found
larsoner commented 5 years ago

Digging a bit more into the code, it look like the 'meas_date' field of the info is not being saved:

In [64]: orig_raw.info['meas_date'] Out[64]: (0, 2147483647)

That meas_date is special, it's meant to be equivalent to None. So it should be fine that it's not saved.

This looks like a dup of #5856 so I'll close, feel free to reopen if I'm wrong

larsoner commented 5 years ago

Actually the other one is an allclose problem and this is a can't-even-read problem, reopening.

fraimondo commented 5 years ago

I just checked using a file < 2GB so it's single. And I still have the same issue.

The file that has the problem comes from GDF files read with read_raw_edf.

Workaround: Set the date to 1970-01-01 00:00:00 before saving

raw.info['meas_date'] = (0, 0)

Here's a code to reproduce the issue:

n_chans = 139
n_samps = 20
data = np.random.random_sample((n_chans, n_samps))
ch_names = ['E{}'.format(x) for x in range(n_chans)]
ch_types = ['eeg'] * n_chans
info = mne.create_info(ch_names=ch_names, ch_types=ch_types, sfreq=2048)
info['meas_date'] = (0, 2147483647)
raw = mne.io.RawArray(data=data, info=info)
raw.save('test-raw.fif', overwrite=True)

raw2 = mne.io.read_raw_fif('test-raw.fif', preload=True)
larsoner commented 5 years ago

Yes we should treat DATE_NONE (i.e., `(0, 2147483647)) the same as None but we appear not to do so.

agramfort commented 5 years ago

indeed

Fede do you have time to send a PR?

fraimondo commented 5 years ago

Yes.

I'll take a look.

What I don't get its if the annotations should not work with DATE_NONE or None should be replaced with DATE_NONE.

In any case, the EDF reader is giving me that date and the annotations.

On Fri, 8 Feb 2019, 21:48 Alexandre Gramfort, notifications@github.com wrote:

indeed

Fede do you have time to send a PR?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/5908#issuecomment-461941719, or mute the thread https://github.com/notifications/unsubscribe-auth/AESRg81N6JMUwJB5Qh7OPUOlshYuKEiuks5vLeKLgaJpZM4awZsZ .

larsoner commented 5 years ago

Thinking about it again, we should detect meas_date == DATE_NONE on reading FIF and set meas_date = None, in other words this is really a problem with our FIF I/O somewhere.

larsoner commented 5 years ago

Argh actually we have explicit tests about having DATE_NONE in there, so never mind. Let's just make Annotations treat None and DATE_NONE the same way.

agramfort commented 5 years ago

yes

can you open the PR or you want to let this to @massich ?

larsoner commented 5 years ago

@fraimondo I think that's a question to you -- can you try, and if it does not make sense, someone else can take a stab at it?

fraimondo commented 5 years ago

Let me take a look.

So for Annotations, None and DATE_NONE should be the same.

I’ll see how this could work with my GDF files with annotations and bad dates. As it is now, annotations should not work without a not None meas_date.

On 10 Feb 2019, at 22:09, Eric Larson notifications@github.com wrote:

@fraimondo https://github.com/fraimondo I think that's a question to you -- can you try, and if it does not make sense, someone else can take a stab at it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/5908#issuecomment-462173139, or mute the thread https://github.com/notifications/unsubscribe-auth/AESRg6QsnbTYq2UJXXFvTyl4S4Z77Iq9ks5vMIqUgaJpZM4awZsZ.

massich commented 5 years ago

@fraimondo if you have any trouble with it you can ping me. (or if you prefer it I can address it in #5857)

larsoner commented 5 years ago

@fraimondo still interested in giving this a shot? If not @massich or I can try

fraimondo commented 5 years ago

I’m super overwhelmed right now.

I can test it, but I don’t have time now to come with a solution. Sorry.

On 27 Mar 2019, at 18:33, Eric Larson notifications@github.com wrote:

@fraimondo https://github.com/fraimondo still interested in giving this a shot? If not @massich https://github.com/massich or I can try

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/5908#issuecomment-477270044, or mute the thread https://github.com/notifications/unsubscribe-auth/AESRgwyTFlVkaCbU20XWmDSayRLj_QGpks5va6tRgaJpZM4awZsZ.

larsoner commented 5 years ago

I’m super overwhelmed right now.

I'll take a look soon then and ping you for a test

massich commented 5 years ago

@larsoner I can take it, if you want

larsoner commented 5 years ago

@massich feel free. Might be best to take care of #5857 and #5852 first

larsoner commented 5 years ago

@fraimondo I adapted this into a mne/io/tests/test_meas_info.py test:

def test_date_none(tmpdir):
    """Test that DATE_NONE is used properly."""
    # Adapted from gh-5908
    n_chans = 139
    n_samps = 20
    data = np.random.random_sample((n_chans, n_samps))
    ch_names = ['E{}'.format(x) for x in range(n_chans)]
    ch_types = ['eeg'] * n_chans
    info = create_info(ch_names=ch_names, ch_types=ch_types, sfreq=2048)
    info['meas_date'] = (0, 2147483647)
    raw = RawArray(data=data, info=info)
    fname = op.join(str(tmpdir), 'test-raw.fif')
    raw.save(fname)
    read_raw_fif(fname, preload=True)

It does not fail. Neither does your minimal code snippet above. So I think this has been fixed (probably by a recent Annotations PR). Feel free to reopen if I'm mistaken.

massich commented 5 years ago

This works for me in v0.16 (onwards):


~/code/mne-python master*
(mne) ❯ git checkout v0.16
Note: checking out 'v0.16'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at ff5dd47a4 update version numbers

~/code/mne-python tags/v0.16*
(mne) ❯ ipython
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import mne                                                                               

In [2]: mne.sys_info()                                                                           
Qt WebEngine seems to be initialized from a plugin. Please set Qt::AA_ShareOpenGLContexts using QCoreApplication::setAttribute before constructing QGuiApplication.
Platform:      Linux-4.15.0-46-generic-x86_64-with-debian-buster-sid
Python:        3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)  [GCC 7.3.0]
Executable:    /home/sik/miniconda3/envs/mne/bin/python
CPU:           x86_64: 4 cores
Memory:        31.3 GB

mne:           0.16.0
numpy:         1.14.3 {blas=openblas, lapack=openblas}
scipy:         1.2.1
matplotlib:    3.0.3 {backend=Qt5Agg}

sklearn:       0.20.3
nibabel:       2.4.0
mayavi:        4.7.0.dev0 {qt_api=pyqt5}
pycuda:        Not found
skcuda:        Not found
pandas:        0.24.2

In [3]: fname = mne.datasets.sample.data_path() + '/MEG/sample/sample_audvis_raw.fif' 
   ...: raw = mne.io.read_raw_fif(fname) 
   ...: raw = mne.concatenate_raws([raw]) 
   ...: raw.save('test_raw.fif', overwrite=True) 
   ...: raw_read = mne.io.read_raw_fif('test_raw.fif')  # this breaks                            

In [4]:                                                                                          ```