BUG: EDF stim resampling when NaN/inf are present

[Andrea-V]

Describe the bug

I cannot plot the data again after applying filtering.

Steps to reproduce

    data_highpass = data.copy().filter(l_freq=0.5, h_freq=None)

    data.plot(n_channels=61, scalings="auto") # this works as expected
    plt.show()

    data_highpass.plot(n_channels=61, scalings="auto") # here I get the error
    plt.show()

Expected results

It should be able to plot data after filtering

Actual results

ValueError: cannot convert float NaN to integer

Additional information

Platform: Linux-5.3.0-45-generic-x86_64-with-glibc2.10 Python: 3.8.1 (default, Jan 8 2020, 22:29:32) [GCC 7.3.0] Executable: /home/andrea/anaconda3/envs/bci_competition/bin/python CPU: x86_64: 8 cores Memory: 7.6 GB

mne: 0.19.2 numpy: 1.18.1 {blas=mkl_rt, lapack=mkl_rt} scipy: 1.4.1 matplotlib: 3.2.0 {backend=TkAgg}

sklearn: 0.22.2.post1 numba: Not found nibabel: Not found cupy: Not found pandas: 1.0.1 dipy: Not found mayavi: Not found pyvista: Not found vtk: Not found

[agramfort]

it seems filtering creates NaN

[larsoner]

@Andrea-V we can't tell from the last line of the traceback where the error is actually happening. Can you paste the full traceback?

[Andrea-V]

here is the full error:

Traceback (most recent call last):
  File "preprocessing_upper_limb.py", line 54, in <module>
    data_highpass.plot(n_channels=61, scalings="auto")      

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/io/base.py", line 1562, in plot
    return plot_raw(self, events, duration, start, n_channels, bgcolor,

File "</home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/externals/decorator.py:decorator-gen-132>", line 2, in plot_raw

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/utils/_logging.py", line 90, in wrapper
    return function(*args, **kwargs)

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/raw.py", line 458, in plot_raw
    callback_proj('none')

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/utils.py", line 334, in _toggle_proj
    params['plot_update_proj_callback'](params, bools)

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/raw.py", line 43, in _plot_update_raw_proj
    params['plot_fun']()

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/raw.py", line 827, in _plot_raw_traces
    '%s %s ' % (_simplify_float(inv_norm), units),

File "/home/andrea/anaconda3/envs/bci_competition/lib/python3.8/site-packages/mne/viz/utils.py", line 523, in _simplify_float
    if isinstance(label, float) and float(str(label)) != round(label):

ValueError: cannot convert float NaN to integer

[larsoner]

Can you see if https://github.com/mne-tools/mne-python/pull/7583 at least does not die?

Then we can figure out why filtering the data has caused nan to show up everywhere.

[larsoner]

@Andrea-V if you can share the data via DropBox or other link, it would also help

[Andrea-V]

You can find a data sample here: https://drive.google.com/drive/folders/1DKs5xPE_DfFbCaIToBN9F3UYnxf5s9Hf?usp=sharing

This is the code I use for loading the data data = mne.io.read_raw_gdf(os.path.join(raw_path, file), verbose=True) data.load_data()

[larsoner]

>>> np.isfinite(raw.get_data()).all()
False

So it appears as though your data already have NaN (filtering will just spread these). The fix might just be to fix the auto-scaling to account for nan values, I'll look.

[Andrea-V]

is there a way to deal with NaN directly into the Raw object? It would be nice to have methods similar to pandas .dropna() or fillna() methods

[Andrea-V]

Or else, a way to create the Raw object from a pandas DataFrame, so that I could deal first with NaN in pandas, load the data into Raw object, and then apply some filtering

BTW: thank you for your timely responses!

[larsoner]

You can create Raw objects from NumPy arrays

https://mne.tools/stable/auto_examples/io/plot_objects_from_arrays.html#sphx-glr-auto-examples-io-plot-objects-from-arrays-py

[larsoner]

It looks like this file is particularly problematic, as there are channels that need to be resampled, and those channels have NaN or inf in them. I've added plotting fixes to #7583, but really we need to deal with EDF resampling better somehow (replace with linear interpolated values or something?)

[christian-oreilly]

is there a way to deal with NaN directly into the Raw object? It would be nice to have methods similar to pandas .dropna() or fillna() methods

For reference, this works for me...

def fillna(raw, fill_val=0):
    return mne.io.RawArray(np.nan_to_num(raw.get_data(), nan=fill_val), raw.info)