step_size _fit_eeg_distribution

bertrandlalo commented 4 years ago

It is still not clear for me how to choose this parameter step_size in _fit_eeg_distribution
Actually, one cannot choose any of the parameter from this function since it is call in fit with the default ones.
For example, I tried to fit a 1 min EEG data sampled at 256Hz with kwargs: {"window_len": 3, "window_overlap": 0.66, "rejection_cutoff": 3} and got the following error. I guess if I had a larger chunk to fit, it'll be ok, but still, it is not easy for the user to guess how many samples he needs, and it seems to depend on the window_len he chooses. Could you please, either fix it, or explain what's the relationship between minimum calibration duration and window_len?
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-17-e5c0cad83d85> in <module>
 18 rate= 256
 19 df_fit = df_filtered[begins:ends]
---> 20 rASR_pipeline = rASR_pipeline.fit( df_fit.values)
 21 
 22 # transform all session
```

~/miniconda3/envs/timeflux-tutorials/lib/python3.7/site-packages/sklearn/pipeline.py in fit(self, X, y, fit_params) 352 self._log_message(len(self.steps) - 1)): 353 if self._final_estimator != 'passthrough': --> 354 self._final_estimator.fit(Xt, y, fit_params) 355 return self 356

~/PycharmProject/OpenMindInnovation/timeflux/timeflux_rasr/timeflux_rasr/estimation.py in fit(self, X, y) 183 184 for c in range(Ne): --> 185 dist_params[c, :] = _fit_eeg_distribution(rmssliding[:, c]) 186 self.threshold = np.diag(dist_params[:, 0] + self.rejection_cutoff * dist_params[:, 1]).dot( 187 np.transpose(evecs))

~/PycharmProject/OpenMindInnovation/timeflux/timeflux_rasr/timeflux_rasr/estimation.py in _fit_eeg_distribution(X, min_clean_fraction, max_dropout_fraction, quantile_range, step_sizes, beta_range) 357 raise ValueError('Unreasonable step sizes.') 358 if any(step_sizes n < 1): --> 359 raise ValueError(f"Step sizes compared to actual number of samples available, step_sizes n should be " 360 f"greater than 1 (current value={step_sizes * n}") 361 if any(beta_range >= 7) | any(beta_range <= 1):

ValueError: Step sizes compared to actual number of samples available, step_sizes * n should be greater than 1 (current value=[0.29 0.29]

lkorczowski commented 4 years ago

I'm looking into it.

lkorczowski commented 4 years ago

@bertrandlalo should be fixed, the parameters in RASR matters now. Sorry for this huge mistake. Please review #21 and valid PR if it is ok for you.

Please note than 1min is probably barely enough to have hundreds of intervals required in _fit_eeg_distribution(). With window_len is 3s and window_overlap 0.66, you should have about 29 epochs while we need at least 100 epochs in _fit_eeg_distribution. With window_len is 0.5s and window_overlap 0.66, you should have about 199 epochs which is just enough to estimate consistent statistics in _fit_eeg_distribution.

Considering at least, 1min30 (299 epochs) or 2min (399 epochs) should be much better.

if window_len is 3s, 5min (149 epochs) with 0.66 of overlap is minimum or you can increase window_overlap to 0.9 (191 epochs) with 1min.

lkorczowski commented 4 years ago

Also, in #21 , I've corrected window_overlap to match description and added sanity check.

lkorczowski commented 4 years ago

For Sprint3 (#15 ):

added task to right better doc and warnings/assert when not enough data is available for training
while the 2D compatibility will be dropped, it will be the user task to epoch the data such as there is enough windows (I believe we should be a minimum of 150 epochs).

Is that a good strategy ?

timeflux / timeflux_rasr

step_size _fit_eeg_distribution #19