Using this package as a plugin in BIDSCoin

[yjmantilla]

I take this verbatim from INCF thread

Hi there, I noticed your interest in BIDScoin (of which I’m the main author) :slight_smile: You may be interested to know I took the idea of plugins to a higher level and now abstracted the GUI and DICOM-conversion away from each other, and now also implemented the dcm2niix conversion in plugins myself (an example of eat-your-own-dogfood). I think EEG conversion could be implemented as a plugin too and would be happy to support that and provide tips and help. All the best, Marcel

So after the package is done maybe there is a way for it to plug into BIDSCoin 🤔

[yjmantilla]

Notes about the possible collaboration/integration with BIDScoin.

Write a short description of how such a collaboration work?

Basically BIDSCOIN has defined already an ontology for plugins. It consist of four elements:

• is_sourcefile: A routine to assess whether the file is of a valid dataformat for this plugin
• get_attribute: A routine for reading an attribute from a sourcefile
• bidsmapper_plugin: A routine that can be called by the bidsmapper to make a bidsmap of the source data
• bidscoiner_plugin: A routine that can be called by the bidscoiner to convert the source data to bids

From this what I think is that:

• get_attribute should be equivalent to the applying a single rule in our ontology.

• bidsmapper_plugin should be equivalent to the whole APPLY RULES module

• bidscoiner_plugin should be equivalent to the CONVERT THEM module

So basically there should be a way to map our API to the BIDSCOIN plugin API. Now, I think we need to work first on our API and make the goal of it being implemented in BIDSCOIN a secundary one.

Now, the DISCOVER RULES module is the problem, that would need further investigation. In the worst case we would have some predefined rules for some common metadata formats that BIDSCOIN can use. More custom cases would require seeing how to translate the DISCOVER RULES module to the BIDSCOIN software ontology.

reference

how will it be changing what you put in the proposal (if at all)

I don't think it would change the proposal as a whole. Mainly we should inspire our API in the example already set by the BIDSCOIN plugin.

And what will be the main challenges?

• The DISCOVER RULES module is what bugs me. 🤔

Nevertheless, the overall design of BIDSCOIN seems similar to what we designed:

• bidsmapper = DISCOVER RULES+APPLY RULES BACKEND
• bidseditor = user-directed loop of APPLY RULES
• bidscoiner = CONVERT THEM module.

So, intuitively there should be a mapping between these "mappers" 😆

• Now, the GUI part is not entirely clear to me how it would work. Somehow it appears to me that such part is abstracted away from the plugins. I will have to ask Marcel Zwiers how does that work. I will do this as an issue inside the bidscoin repo https://github.com/Donders-Institute/bidscoin/issues/77

What will be the main benefits, other than faster development time and the leveraging of future contributions from the BIDScoin development team?

I think mostly is about expanding the proposed back-end usage to many BIDS converters. In a way is about centralizing EEG/MEG bids conversion efforts to a single repo so that development is of interest to many teams.

In a sense, this is the same advantage you pointed out.

What are the main disadvantages?

Hmmm, from the disadvantage point of view I would guess is the time-investment around the GUI part. Since we can always just support some metadata formats in bidscoin and not the custom formats the DISCOVER RULES wouldnt be always necesary.

Most importantly, I understand that BIDScoin is GUI-based, but not web. Is that the case, or do the python graphic libraries they use (forgot which those are) allow easy migration to a web-based GUI, based on the same codebase?

I did a quick skim-search and I don't think there is an easy pyQt5 to web migration. This is what I intuitively expected since webapps after all are another kind of beast apart from desktop apps. I would argue it would be actually easier to plug a webapp into pyQt5 via the web engine and browser they have.

I think we are better of functioning as a backend of the BIDSCOIN plugin and leave the GUI to the respective softwares that we are plugged into.

[civier]

Very nice work Yorguin. You really delved into that.

Will it be correct to say that BIDScoin is the only tool out there that not only has GUI for conversion, but the GUI is abstracted away from the actual conversion? If yes, so it's really a bargain, as long as the API / communication between these two components will be suitable for the GUI we want to have for the MEEG converter. However, if the GUI for MEEG conversion is drastically different from that used for DICOM conversion, so maybe the BIDScoin's GUI-to-conversion API will not work for us, and then there is no point really. What do you think?

I also want to understand what Marcel means exactly by "abstracted the GUI and DICOM-conversion away from each other"? If we want to interface with web-based GUIs / non-python GUIs, we need an actual API between these two components (or a two-way communication based on files, pipeline, etc.). Is that the case in BIDScoin? Or is it that the "abstraction" is just within Python, i.e. separate objects/functions/libraries? Maybe you can clarify it when you speak with him.

Best, Oren

[yjmantilla]

@civier

Will it be correct to say that BIDScoin is the only tool out there that not only has GUI for conversion, but the GUI is abstracted away from the actual conversion?

My intuition is that this is true, but we would need to do a characterization of all the tools we know. We kind of started doing that sometime ago but we didn't continue it.

You can check it here

However, if the GUI for MEEG conversion is drastically different from that used for DICOM conversion, so maybe the BIDScoin's GUI-to-conversion API will not work for us, and then there is no point really. What do you think?

From what Marcel told me in https://github.com/Donders-Institute/bidscoin/issues/77 , what I understand is that:

• Yes, GUI is abstracted away from the plugins
• We can sort of configure the GUI through a YAML file. He showed an example in the issue I opened.

So the question is: Can we configure BIDSCOIN GUI in such a way that works for sovabids?

I also want to understand what Marcel means exactly by "abstracted the GUI and DICOM-conversion away from each other"? If we want to interface with web-based GUIs / non-python GUIs, we need an actual API between these two components (or a two-way communication based on files, pipeline, etc.). Is that the case in BIDScoin? Or is it that the "abstraction" is just within Python, i.e. separate objects/functions/libraries? Maybe you can clarify it when you speak with him.

I will need to meditate this further...

[TomEmotion]

Conda MNE environment for the VNM - just change the filename to mne.yml (github doesn't support yml files).

mne_yml.txt

[civier]

Thank you @TomEmotion

@yjmantilla Notice that the VNM was developed with data analysis in mind, so it might not include many tools for actual code development. That said, you can install within the VNM whatever you need. If those are small utilities that should be included in the base image in your opinion, please let us know on the Discussions tab of NeuroDesk (https://github.com/NeuroDesk/neurodesk/discussions), and we will make sure to include them in the next release. Notice that some utilities are already included. You can find the list here: https://github.com/NeuroDesk/vnm/#list-of-available-software-in-the-desktop-no-need-to-download-already-included-in-main-container

If you want to take part in the development of NeuroDesk, you're most welcome to submit issues or work on some. Just keep in mind that the Discussions and Issues tabs we use are under https://github.com/NeuroDesk/neurodesk Because VNM is a sub-project of NeuroDesk, we prefer to keep all tickets in the latter one.

[yjmantilla]

@civier

I also want to understand what Marcel means exactly by "abstracted the GUI and DICOM-conversion away from each other"? If we want to interface with web-based GUIs / non-python GUIs, we need an actual API between these two components (or a two-way communication based on files, pipeline, etc.). Is that the case in BIDScoin? Or is it that the "abstraction" is just within Python, i.e. separate objects/functions/libraries? Maybe you can clarify it when you speak with him.

From what I understood reading Marcel posts and the code itself it seems it is a two-way communication based on files. From https://github.com/Donders-Institute/bidscoin/blob/master/bidscoin/bidseditor.py we have for the gui editor the following description:

This tool launches a graphical user interface for editing the bidsmap that is produced by the bidsmapper. You can edit the BIDS data types and entities until all run-items have a meaningfuland nicely readable BIDS output name. The (saved) bidsmap.yaml output file will be used by thebidscoiner to do the conversion conversion of the source data to BIDS.

We have then,

bidsmapper --> bidsmap --> gui editor --> bidsmap_edited ---> bidscoiner

Thats why the plugins are not themselves connected to the gui editor (except for some particular cases where they are able to edit the configuration of the editor or something like that).

Translated to our design this is equivalent to the loop in the "APPLY RULES" module. This module produces a mapping file that would be editable by the bidseditor interface and would then produce an edited mapping file. The difference is that our "APPLY RULES" module grabs a "RULES FILE" to produce the initial guessed mapping. In bidscoin this is done internally by the "bidsmapper" and it does not produce a file; my guess is that such information is inside heuristics.py .

[yjmantilla]

@TomEmotion @civier the environment is quite large, is there a problem with me installing the default mne environment ?

Right now I'm trying to install the yml you gave me but there are some conflicts:

The next environment solving also returned conflicts, so I stopped it.

PD: I was able to use the NeuroDesk VNM. Quite impressive.

[stebo85]

Dear @yjmantilla ,

great work already in getting this project started!

I agree that it's best to start with BIDScoin and develop an EEG/MEG plugin for it.

Also, using the default mne environment should not have the conflicts you are seeing :)

Cheers Steffen

[civier]

AFAIK @TomEmotion included a few extra packages in the environment in addition to MNE-python. @TomEmotion, could you please list the packages you've added, so Yorguin can decide if they are necessary to our project or not? If the standard MNE-python Conda is sufficient, I think we should favour it, as it is probably maintained by the MNE team, and would be updated when new MNE versions are released.

[TomEmotion]

Very sorry - yes, you're correct, I installed some extra packages. They should not be necessary, so please go ahead with the standard MNE environment.

[yjmantilla]

Here some updates I managed to do on this integration with the OHBM Brainhack:

Did a fork of bidscoin to be able to change it a bit:

• https://github.com/yjmantilla/bidscoin/tree/sovabids

I managed to both infer attributes from the files and write them, the example use-case is here. The bidscoin GUI seems to work well with what I did.

https://github.com/yjmantilla/sovabids/blob/main/docs/sovabids_and_bidscoin.md

[stebo85]

That's great progress :) wow 😲

[civier]

I cannot say that I follow everything (need to play with it a bit), but it looks very impressive. @yjmantilla Since by now you know BIDScoin very well, I wonder if you will be able to give us a short tutorial on its architecture, and how you integrated with it. I think it will help us understand what you're doing (especially Tom and I, I think Steffen and Aswin already have experience with BIDScoin) Can we schedule something for early in the week starting 12th of July? In that week we need to submit the first evaluation, so will be good to understand what you're doing in details.

[yjmantilla]

@civier

Since by now you know BIDScoin very well, I wonder if you will be able to give us a short tutorial on its architecture, and how you integrated with it. I think it will help us understand what you're doing (especially Tom and I, I think Steffen and Aswin already have experience with BIDScoin)

Well, I definitively know it better than before 😆 , wouldn't know if that qualifies to "very well". Last week I opened a bunch of issues in bidscoin and Marcel updated the repo so I need to update my repo accordingly since some of the stuff I proposed was included in one way or another. Nicely, I was able to implement a test for the bidscoin conversion.

Can we schedule something for early in the week starting 12th of July? In that week we need to submit the first evaluation, so will be good to understand what you're doing in details.

Yes, what day of that week?

Given this I will focus on the sovabids plugin more (what I did was very hacky, sovabids not being used as I wished). I will work on that.

[yjmantilla]

Hi everyone, I finally finished the update sova2coin plugin.

Here is the example, in case @DavidjWhite33 has time to do it.