BUG: EDF stim resampling when NaN/inf are present

[Andrea-V]

Describe the bug

I cannot plot the data again after applying filtering.

Steps to reproduce

    data_highpass = data.copy().filter(l_freq=0.5, h_freq=None)

    data.plot(n_channels=61, scalings="auto") # this works as expected
    plt.show()

    data_highpass.plot(n_channels=61, scalings="auto") # here I get the error
    plt.show()

Expected results

It should be able to plot data after filtering

Actual results

ValueError: cannot convert float NaN to integer

Additional information

Platform: Linux-5.3.0-45-generic-x86_64-with-glibc2.10 Python: 3.8.1 (default, Jan 8 2020, 22:29:32) [GCC 7.3.0] Executable: /home/andrea/anaconda3/envs/bci_competition/bin/python CPU: x86_64: 8 cores Memory: 7.6 GB

mne: 0.19.2 numpy: 1.18.1 {blas=mkl_rt, lapack=mkl_rt} scipy: 1.4.1 matplotlib: 3.2.0 {backend=TkAgg}

sklearn: 0.22.2.post1 numba: Not found nibabel: Not found cupy: Not found pandas: 1.0.1 dipy: Not found mayavi: Not found pyvista: Not found vtk: Not found

[agramfort]

it seems filtering creates NaN

[larsoner]

@Andrea-V we can't tell from the last line of the traceback where the error is actually happening. Can you paste the full traceback?

[Andrea-V]

here is the full error:

Traceback (most recent call last):
  File "preprocessing_upper_limb.py", line 54, in <module>
    data_highpass.plot(n_channels=61, scalings="auto")      

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/io/base.py", line 1562, in plot
    return plot_raw(self, events, duration, start, n_channels, bgcolor,

File "</home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/externals/decorator.py:decorator-gen-132>", line 2, in plot_raw

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/utils/_logging.py", line 90, in wrapper
    return function(*args, **kwargs)

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/raw.py", line 458, in plot_raw
    callback_proj('none')

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/utils.py", line 334, in _toggle_proj
    params['plot_update_proj_callback'](params, bools)

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/raw.py", line 43, in _plot_update_raw_proj
    params['plot_fun']()

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/raw.py", line 827, in _plot_raw_traces
    '%s %s ' % (_simplify_float(inv_norm), units),

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/utils.py", line 523, in _simplify_float
    if isinstance(label, float) and float(str(label)) != round(label):

ValueError: cannot convert float NaN to integer

[larsoner]

Can you see if https://github.com/mne-tools/mne-python/pull/7583 at least does not die?

Then we can figure out why filtering the data has caused nan to show up everywhere.

[larsoner]

@Andrea-V if you can share the data via DropBox or other link, it would also help

[Andrea-V]

You can find a data sample here: https://drive.google.com/drive/folders/1DKs5xPE_DfFbCaIToBN9F3UYnxf5s9Hf?usp=sharing

This is the code I use for loading the data data = mne.io.read_raw_gdf(os.path.join(raw_path, file), verbose=True) data.load_data()

[larsoner]

>>> np.isfinite(raw.get_data()).all()
False

So it appears as though your data already have NaN (filtering will just spread these). The fix might just be to fix the auto-scaling to account for nan values, I'll look.

[Andrea-V]

is there a way to deal with NaN directly into the Raw object? It would be nice to have methods similar to pandas .dropna() or fillna() methods

[Andrea-V]

Or else, a way to create the Raw object from a pandas DataFrame, so that I could deal first with NaN in pandas, load the data into Raw object, and then apply some filtering

BTW: thank you for your timely responses!

[larsoner]

You can create Raw objects from NumPy arrays

https://mne.tools/stable/auto_examples/io/plot_objects_from_arrays.html#sphx-glr-auto-examples-io-plot-objects-from-arrays-py

[larsoner]

It looks like this file is particularly problematic, as there are channels that need to be resampled, and those channels have NaN or inf in them. I've added plotting fixes to #7583, but really we need to deal with EDF resampling better somehow (replace with linear interpolated values or something?)