epochs repr is different if you do epochs.drop_bad instead of Epochs(..., reject)

[jasmainak]

The code will explain itself:

import mne

from mne.datasets import sample

reject = dict(grad=4000e-13, mag=4e-12, eog=150e-6)

data_path = sample.data_path()
raw_fname = data_path + '/MEG/sample/sample_audvis_filt-0-40_raw.fif'
raw = mne.io.read_raw_fif(raw_fname)

events = mne.find_events(raw, stim_channel='STI 014')
event_id = {"auditory/left": 1}
tmin = -0.2
tmax = 0.5
baseline = (None, 0)
picks_meg = mne.pick_types(raw.info, meg=True, eeg=False, eog=True,
                           stim=False, exclude='bads')
epochs = mne.Epochs(raw, events, event_id, tmin, tmax, proj=True,
                    picks=picks_meg, baseline=baseline, reject=None)
epochs.drop_bad(reject=reject)
print(epochs)

The repr says: 55 events (all good)

even though 17 epochs were dropped.

[agramfort]

I don't get the pb

In [6]: %run debug
Opening raw data file /Users/alex/mne_data/MNE-sample-data/MEG/sample/sample_audvis_filt-0-40_raw.fif...
    Read a total of 4 projection items:
        PCA-v1 (1 x 102)  idle
        PCA-v2 (1 x 102)  idle
        PCA-v3 (1 x 102)  idle
        Average EEG reference (1 x 60)  idle
    Range : 6450 ... 48149 =     42.956 ...   320.665 secs
Ready.
Current compensation grade : 0
319 events found
Event IDs: [ 1  2  3  4  5 32]
72 matching events found
Applying baseline correction (mode: mean)
Not setting metadata
Created an SSP operator (subspace dimension = 3)
4 projection items activated
Loading data for 72 events and 106 original time points ...
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on MAG : ['MEG 1711']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EOG : ['EOG 061']
17 bad epochs dropped
<Epochs  |   55 events (all good), -0.199795 - 0.499488 sec, baseline [None, 0], ~3.5 MB, data not loaded,
 'auditory/left': 55>

In [7]: epochs.events.shape
Out[7]: (55, 3)

[mmagnuski]

@agramfort You do see it towards the end:

17 bad epochs dropped
<Epochs  |   55 events (all good), -0.199795 - 0.499488 sec, baseline [None, 0], ~3.5 MB, data not loaded,
 'auditory/left': 55>

but that depends on whether you interpret this repr as being a problem or not. I actually wondered a few times when I could get something other than (all good) - I don't think I ever got a different description even though I was rejecting epochs in a variety of ways.

[larsoner]

I actually wondered a few times when I could get something other than (all good) - I don't think I ever got a different description even though I was rejecting epochs in a variety of ways.

Just remove the epochs.drop_bad(reject=reject) line from the example and you'll see it:

<Epochs  |   72 events (good & bad), -0.199795 - 0.499488 sec, baseline [None, 0], ~3.5 MB, data not loaded,
 'auditory/left': 72>

And for one that's more relevant, you can have reject=reject in the Epochs constructor (just keep preload=False) and you'll get the same repr with (good & bad), but doing load_data() or drop_bad will change it to (all good):

>>> epochs = mne.Epochs(raw, events, event_id, tmin, tmax, proj=True,
...                     picks=picks_meg, baseline=baseline, reject=reject)
>>> print(epochs)
<Epochs  |   72 events (good & bad), -0.199795 - 0.499488 sec, baseline [None, 0], ~3.5 MB, data not loaded,
 'auditory/left': 72>
>>> epochs.load_data()
>>> print(epochs)
<Epochs  |   55 events (all good), -0.199795 - 0.499488 sec, baseline [None, 0], ~17.1 MB, data loaded,
 'auditory/left': 55>

[mmagnuski]

@larsoner Ok, thanks, that helps. But how does it know in the example if you don't do drop_bad that there are good & bad events (although Epochs was not constructed with rej param)? Does preloading the data (still without reject param) changes the state of good & bad to all good? I will check myself, but I am waiting for the sample data to download and that will take some time. :)

[mmagnuski]

(this would explain why I don't remember ever seeing anything else than all_good because I rarely used rej when constructing Epochs and always preloaded the data)

[larsoner]

But how does it know in the example if you don't do drop_bad that there are good & bad events

It's more of "maybe some good maybe some bad" rather than "definitely some good and definitely some bad". It's almost a misnomer with reject=None, except that some can still be rejected due to annot, TOO_SHORT, etc.

[larsoner]

And yes loading data should always change state to all good

[mmagnuski]

Ok, this was my point of confusion, then it's clear for me now (and I guess to @jasmainak too - as this seems to have been the point of confusion in his case too - as drop_bad loads the data). That might be worth documenting somewhere if its not already explained in the docs (the interaction with preload might be confusing). But otherwise

[jasmainak]

right, I think I got confused because Epochs(..., reject=reject) and epochs.drop_bad(reject=reject) appear to do the same thing but they don't.

Epochs(..., reject=reject) just declares an intention to reject (and the rejection is done when the data is loaded/accessed) whereas epochs.drop_bad(reject=reject) reject right away because it loads the data. The fact that I coded parts of these functions myself didn't seem to have helped :)

[jasmainak]

I'm closing this as it seems to be a false alarm