Store the scores alongside the components chosen by find_bads_eXg methods.

mscheltienne commented 3 years ago

Following a discussion on the forum with @hoechenberger I could not find a way to recover the information I was looking for.

Following an ICA, I saved to disk both the ICA instance and the raw instance on which the ICA is applied.

ica = mne.preprocessing.ICA(method='picard', max_iter='auto', random_state=101)
ica.fit(raw, picks='eeg')
eog_idx, eog_scores = ica.find_bads_eog(raw)
ecg_idx, ecg_scores = ica.find_bads_ecg(raw)
ica.exclude = eog_idx + ecg_idx
ica.apply(raw)

ica.save()
raw.save()

I wanted to check how well the adaptative Z-score threshold worked (I like smart algo, but I don't trust them blindly). After posting on the forum, and looking for a while, I could retrieve the separation between excluded components corresponding to EOG-related activity and ECG-related activity. This information is stored in ica.labels_. However, I could not recover the scores that lead to this component selection.

I ended up rerunning part of my pipeline and saving the scores separately, but it was very surprising for me that I could not recover this information from the ICA instance. After looking a bit into the code, it was clear to me that the scores were not saved anywhere. I propose to add an additional attribute similar to labels_ but for the scores.

agramfort commented 3 years ago

no objection to this but it will need FIFF adjustments I suspect

adam2392 commented 2 years ago

We (@mscheltienne , @jacobf18, @anandsaini024) are setting up https://github.com/jacobf18/iclabel-python to now contain an exact port of the "ICLabel" pipeline in EEGLab, but now in Python. I think this will be an amazing feature addition to the MNE toolkit. Moreover, we are then in a position to improve the model, making component labeling more and more an automated process.

We're interested in pursuing this Issue to tighten up the integration of mne-icalabel with mne-python. Our plan moving forward is outlined in https://github.com/jacobf18/iclabel-python/issues/19#issuecomment-1112365416.

We are wondering if it will be possible to move mne-icalabel to mne-tools? We will need to rename the repository and such. If so, is it possible to give Mathieu, Jacob and Anand "team-member" access to MNE for only that repository?

adam2392 commented 2 years ago

Also I forgot to say: the current tests "demonstrate" that the pipeline exactly matches that of EEGLab. Sorting out API kinks. Perhaps we can discuss in the core-dev meeting sometime.

agramfort commented 2 years ago

no real objection

can you do a pass on readme (make sure links work), put a doc and examples online?

Message ID: @.***>

adam2392 commented 2 years ago

https://adam2392.github.io/mne-icalabel/dev/index.html

Is the preliminary docs, and everything else should link up to mne.tools instead of adam2392. Gonna sort out a short example.

adam2392 commented 2 years ago

Not sure why the example is not rendering, but the docs are consolidated now. Here is an example in action: https://output.circle-artifacts.com/output/job/7cd5cc2f-d66a-4644-a428-9dba14f6c765/artifacts/0/dev/auto_examples/iclabel_automatic_artifact_correction_ica.html#sphx-glr-auto-examples-iclabel-automatic-artifact-correction-ica-py

Many links all lead to mne.tools, but will automatically work when it's migrated off of adam2392.github.io.

agramfort commented 2 years ago

If you transfer to me your repo I will then transfer it to mne-tools org

adam2392 commented 2 years ago

Thanks @agramfort ! will do so after making v0.1 release on pypi(?).

Just tidying up the docs/api/datasets.

hoechenberger commented 2 years ago

Thanks @agramfort ! will do so after making v0.1 release on pypi(?).

Once that's done, LMK and I can create a conda-forge package too, so we can include this in the installer!

adam2392 commented 2 years ago

Kay v0.1 up, docs updated and transfer request to @agramfort was sent. Lmk if there's anything missing!

agramfort commented 2 years ago

Ok transfer done

adam2392 commented 2 years ago

Hi,

reviving this thread now that the mne-icalabel API has evolved a bit to expose what is needed on MNE-Python side. Besides storing the scores with an array: labels_scores_, it would be nice to have a final list of possible labels inside labels_ dictionary. Rn there are: ctpsp, ecg, and muscle and ref_mag I think. It would be nice to extend this possible list of candidates and document them in MNE-Python.

Extend and fix the possible values inside ica.labels_ dictionary: My proposal is 'ctpsp', 'ecg', 'brain', 'muscle', line_noise', 'ch_noise', 'ref_mag' and 'other'.
Add labels_scores_ array to the ICA instance which optionally stores a 2D array of scores with columns representing the score for each of the possible values in labels_ and the rows representing the specific component.

WDYT @agramfort, @larsoner @drammock ?

cc: @mscheltienne

agramfort commented 2 years ago

once you store it this way what to expect in terms of usage? this would come with some visualization? the labels could be set automatically or manually?

Message ID: @.***>

adam2392 commented 2 years ago

Once you have them: mne API would allow you to visualize and save to disc.

To get the labels/scores: these would be set using a variety of methods. E.g. right now there's methods to set for ecg using an existing ecg electrode as a template I think. It can also be set automatically with mne-icalabel. And finally, the labels can be set with the ICA labeling GUI as well.

So the label scores is an optional array set by scoring methods.

agramfort commented 2 years ago

ok so np.argmax(labelsscores, axis=1) may be overruled by labels_ with manual adjustment. is that what you mean?

On Mon, Jul 18, 2022 at 10:36 PM Adam Li @.***> wrote:

Once you have them: mne API would allow you to visualize and save to disc.

To get the labels/scores: these would be set using a variety of methods. E.g. right now there's methods to set for ecg using an existing ecg electrode as a template I think. It can also be set automatically with mne-icalabel. And finally, the labels can be set with the ICA labeling GUI as well.

So the label scores is an optional array set by scoring methods.

— Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/9846#issuecomment-1188279762, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABHKHAQU3JXRF3XOA2TW33VUW53NANCNFSM5F7RL34Q . You are receiving this because you were mentioned.Message ID: @.***>

adam2392 commented 2 years ago

I'm inclined to say yes, because you could automatically annotate with a method and then manually double check.

If you automatically annotate with a method then my intuition is that they should match exactly.

agramfort commented 2 years ago

works for me.

Message ID: @.***>

adam2392 commented 2 years ago

Actually an additional issue is that labels_ can store multiple labels per component. For example, I am browsing the examples here: https://mne.tools/stable/auto_tutorials/preprocessing/40_artifact_correction_ica.html

And I see the following.

I am not 100% sure why there would ever be multiple labels for the same component. In addition, would this affect our proposal? I'm guessing no, but not 100% sure. I think I am proposing the following constraints on the labels_ dictionary:

it by default has keys: `['brain', 'ecg', 'eog', 'muscle', 'ch_noise', 'line_noise', 'other'] with empty lists
it can add other entries, such as in the example, one could add a 'blink' key, which might be different from EOG because maybe it's just "one blink" in the artifact rather than an EOG signal?
automated labeling: these methods would append components into the corresponding lists
manual labeling: these methods would append components into the corresponding lists
The default keys are documented in the ICA class.

drammock commented 2 years ago

I am not 100% sure why there would ever be multiple labels for the same component.

it is explained in the very next paragraph after your screenshot:

Notice that the first subject has 3 different labels for the IC at index 0: “eog/0/Fpz”, “eog”, and “blink”. The first two were added by find_badseog; the “blink” label was added by the last call to corrmap. [...] The `labels` attribute of ICA objects can also be manually edited to annotate the ICs with custom labels. They also come in handy when plotting

adam2392 commented 2 years ago

Sorry I meant I'm not 100% sure what the use case then would be. The paragraph states that it is added by methods, but ike if someone/some-method said it was "eog" and "blink", how does that benefit the researcher per say? Or perhaps that's irrelevant question and highly dependent on whatever the researcher is doing?

If this train of thought is irrelevant though for our proposal of:

adding default entries to labels_ and
adding labels_scores_ to ICA

then nvm :p.

drammock commented 2 years ago

I think the use case is going to be researcher-dependent. And IIUC, all you need is to be able to add entries to labels_ (which should be fine, just make sure you don't overwrite any existing entries) and act accordingly when querying labels_ (i.e., don't expect/assume there is just one key in labels_)

mscheltienne commented 2 years ago

IMO, default documented entries would bring more good than harm.

mne-tools / mne-python

Store the scores alongside the components chosen by find_bads_eXg methods. #9846