mne-tools / mne-hfo

Estimate/compute high-frequency oscillations (HFOs) from iEEG data that are BIDS and MNE compatible using a scikit-learn-style API.
http://mne.tools/mne-hfo/
BSD 3-Clause "New" or "Revised" License
13 stars 16 forks source link

Refactor Hilbert to be More Memory Efficient #33

Closed pmyers16 closed 3 years ago

pmyers16 commented 3 years ago

Describe the problem

The current implementation has an internal state that stores a (n_channels x n_times x n_bands) matrix, which ends up eating up 30+GB of RAM if using linear band method and 20+ GB of RAM if using the log band method for a 30 minute clip sampled at 2000 Hz.

Describe your solution

Default to processing each channels data sequentially, parallelizing over bands, and merging in post-processing when array sizes are more manageable.

Describe possible alternatives

The alternative is to keep it as it is since it runs, just inefficiently

pmyers16 commented 3 years ago

Calling fit would have the following flow:

X, y = _check_input_raw(X, None) Check n_jobs sig = X[idx, :] hfo_statistic_arr = self._compute_hfo_statistic(sig) hfo_detection_arr = self._threshold_statistic(hfo_statistic_arr) hfos = self._post_process_ch_hfos(hfo_detection_arr) Then merge channels into chs_hfos

adam2392 commented 3 years ago

This LGTM. ping me when you want me to look at PR.