[x] Changelog has been updated (docs/source/changes.md)
Closes #864
Closes #861
Closes #857
Closes #804
Sample output
```
$ pytest mne_bids_pipeline/ -k ds000248_ica
...
┌────────┬ preprocessing/_06a1_fit_ica ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│12:49:48│ ⏳️ sub-01 run-01 Processing raw data from sub-01_task-audiovisual_run-01_proc-filt_split-01_raw.fif
│12:49:49│ ⏳️ sub-01 run-01 Applying high-pass filter with 1.0 Hz cutoff …
│12:49:50│ ⏳️ sub-01 run-01 Creating task-related epochs …
│12:49:51│ ⏳️ sub-01 Using PTP rejection thresholds: {'mag': 3e-12, 'grad': 3e-10}
│12:49:51│ ⏳️ sub-01 Saving ICA epochs to disk.
│12:49:51│ ⏳️ sub-01 Calculating ICA solution using method: extended_infomax.
│12:50:14│ ⏳️ sub-01 Fit 31 components (explaining 80.2% of the variance) in 109 iterations.
│12:50:14│ ⏳️ sub-01 Saving ICA solution to disk.
│12:50:14│ ⏳️ sub-01 Initializing ICA.fit report HDF5 file
│12:50:29│ ⏳️ sub-01 Saving ICA.fit report: /home/larsoner/mne_data/derivatives/mne-bids-pipeline/ds000248_ica/sub-01/meg/sub-01_task-audiovisual_proc-icafit_report.html
└────────┴ done (42s)
┌────────┬ preprocessing/_06a2_find_ica_artifacts ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│12:50:29│ ⏳️ sub-01 Loading ICA solution
│12:50:30│ ⏳️ sub-01 Creating ECG epochs …
│12:50:32│ ⏳️ sub-01 Performing automated ECG artifact detection …
│12:50:33│ ⏳️ sub-01 Detected 5 ECG-related ICs in 166 ECG epochs.
│12:50:33│ ⏳️ sub-01 Creating EOG epochs …
│12:50:33│ ⏳️ sub-01 Performing automated EOG artifact detection …
│12:50:33│ ⏳️ sub-01 Detected 1 EOG-related ICs in 10 EOG epochs.
│12:50:33│ ⏳️ sub-01 Saving ICA solution and detected artifacts to disk.
│12:51:19│ ⏳️ sub-01 Saving ICA report: /home/larsoner/mne_data/derivatives/mne-bids-pipeline/ds000248_ica/sub-01/meg/sub-01_task-audiovisual_proc-ica+components_report.html
│12:51:19│ ⏳️ sub-01 ICA completed. Please carefully review the extracted ICs in the report sub-01_task-audiovisual_proc-ica+components_report.h5, and mark all components you wish to reject as 'bad' in sub-01_task-audiovisual_proc-ica_components.tsv
└────────┴ done (50s)
...
┌────────┬ preprocessing/_08a_apply_ica ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│12:51:31│ ⏳️ sub-01 Input: sub-01_task-audiovisual_split-01_epo.fif
│12:51:31│ ⏳️ sub-01 Output: sub-01_task-audiovisual_proc-ica_epo.fif
│12:51:32│ ⏳️ sub-01 Rejecting ICs: 0, 1, 4, 9, 14, 20
│12:51:33│ ⏳️ sub-01 Saving reconstructed epochs after ICA.
│12:51:45│ ⏳️ sub-01 Adding ICA to report.
│12:51:57│ ⏳️ sub-01 Saving ICA.apply report: /home/larsoner/mne_data/derivatives/mne-bids-pipeline/ds000248_ica/sub-01/meg/sub-01_task-audiovisual_report.html
│12:51:57│ ⏳️ sub-01 run-01 Writing sub-01_task-audiovisual_run-01_proc-clean_raw.fif …
│12:51:59│ ⏳️ sub-01 run-01 Adding cleaned raw data to report
│12:52:05│ ⏳️ sub-01 run-01 Saving report: /home/larsoner/mne_data/derivatives/mne-bids-pipeline/ds000248_ica/sub-01/meg/sub-01_task-audiovisual_report.html
└────────┴ done (35s)
```
Splits ICA into: 1) fitting + epoch creation, 2) detection of ecg/eog, 3) applying. To do this, two ICAs (and two reports) now get saved: one for the original fit (proc="icafit"), and a second one for after the automated detection.
Updates our docs with some stuff about caching (figured I'd take a stab at it while updating docs for ICA)
Pretty sure this closes #804 by cleaning up run/session etc. a bit but if you think it's related to stuff other than ICA feel free to remove that from the closes lines above @hoechenberger
Fixes a bug where we put _components.tsv in out_files dict. Our caching code looks to make sure that not just the input files but also the output files have the expected hashes. Since it's expected that users will modify this file, it should not be in out_files in the _06 fit/detect step(s), otherwise it will be a cache miss when users modify the file, and the step will re-run and overwrite their changes (!), e.g. on main:
Before merging …
docs/source/changes.md)Closes #864 Closes #861 Closes #857 Closes #804
Sample output
``` $ pytest mne_bids_pipeline/ -k ds000248_ica ... ┌────────┬ preprocessing/_06a1_fit_ica ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │12:49:48│ ⏳️ sub-01 run-01 Processing raw data from sub-01_task-audiovisual_run-01_proc-filt_split-01_raw.fif │12:49:49│ ⏳️ sub-01 run-01 Applying high-pass filter with 1.0 Hz cutoff … │12:49:50│ ⏳️ sub-01 run-01 Creating task-related epochs … │12:49:51│ ⏳️ sub-01 Using PTP rejection thresholds: {'mag': 3e-12, 'grad': 3e-10} │12:49:51│ ⏳️ sub-01 Saving ICA epochs to disk. │12:49:51│ ⏳️ sub-01 Calculating ICA solution using method: extended_infomax. │12:50:14│ ⏳️ sub-01 Fit 31 components (explaining 80.2% of the variance) in 109 iterations. │12:50:14│ ⏳️ sub-01 Saving ICA solution to disk. │12:50:14│ ⏳️ sub-01 Initializing ICA.fit report HDF5 file │12:50:29│ ⏳️ sub-01 Saving ICA.fit report: /home/larsoner/mne_data/derivatives/mne-bids-pipeline/ds000248_ica/sub-01/meg/sub-01_task-audiovisual_proc-icafit_report.html └────────┴ done (42s) ┌────────┬ preprocessing/_06a2_find_ica_artifacts ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │12:50:29│ ⏳️ sub-01 Loading ICA solution │12:50:30│ ⏳️ sub-01 Creating ECG epochs … │12:50:32│ ⏳️ sub-01 Performing automated ECG artifact detection … │12:50:33│ ⏳️ sub-01 Detected 5 ECG-related ICs in 166 ECG epochs. │12:50:33│ ⏳️ sub-01 Creating EOG epochs … │12:50:33│ ⏳️ sub-01 Performing automated EOG artifact detection … │12:50:33│ ⏳️ sub-01 Detected 1 EOG-related ICs in 10 EOG epochs. │12:50:33│ ⏳️ sub-01 Saving ICA solution and detected artifacts to disk. │12:51:19│ ⏳️ sub-01 Saving ICA report: /home/larsoner/mne_data/derivatives/mne-bids-pipeline/ds000248_ica/sub-01/meg/sub-01_task-audiovisual_proc-ica+components_report.html │12:51:19│ ⏳️ sub-01 ICA completed. Please carefully review the extracted ICs in the report sub-01_task-audiovisual_proc-ica+components_report.h5, and mark all components you wish to reject as 'bad' in sub-01_task-audiovisual_proc-ica_components.tsv └────────┴ done (50s) ... ┌────────┬ preprocessing/_08a_apply_ica ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │12:51:31│ ⏳️ sub-01 Input: sub-01_task-audiovisual_split-01_epo.fif │12:51:31│ ⏳️ sub-01 Output: sub-01_task-audiovisual_proc-ica_epo.fif │12:51:32│ ⏳️ sub-01 Rejecting ICs: 0, 1, 4, 9, 14, 20 │12:51:33│ ⏳️ sub-01 Saving reconstructed epochs after ICA. │12:51:45│ ⏳️ sub-01 Adding ICA to report. │12:51:57│ ⏳️ sub-01 Saving ICA.apply report: /home/larsoner/mne_data/derivatives/mne-bids-pipeline/ds000248_ica/sub-01/meg/sub-01_task-audiovisual_report.html │12:51:57│ ⏳️ sub-01 run-01 Writing sub-01_task-audiovisual_run-01_proc-clean_raw.fif … │12:51:59│ ⏳️ sub-01 run-01 Adding cleaned raw data to report │12:52:05│ ⏳️ sub-01 run-01 Saving report: /home/larsoner/mne_data/derivatives/mne-bids-pipeline/ds000248_ica/sub-01/meg/sub-01_task-audiovisual_report.html └────────┴ done (35s) ```proc="icafit"), and a second one for after the automated detection.closeslines above @hoechenbergerFixes a bug where we put
_components.tsvinout_filesdict. Our caching code looks to make sure that not just the input files but also the output files have the expected hashes. Since it's expected that users will modify this file, it should not be inout_filesin the_06fit/detect step(s), otherwise it will be a cache miss when users modify the file, and the step will re-run and overwrite their changes (!), e.g. onmain:on this PR: