ENH/IO: add support for EGI Netstation EEG data.

[dengemann]

We're currently using EGI EEG together with Neuromag data. I'd like to merge the EGI data into a fiff file. I haven't explored the existing solutions for reading EGI data. Here's what needs to be done.

• define free fiff codes
• make a decision regarding reading the .raw files VS matlab files
• maybe first do a BaseRaw ...

cc @agramfort

[dengemann]

It seems the .raw format would be really easy to support. export to EDF+ might also work.

http://fieldtrip.fcdonders.nl/getting_started/egi

[agramfort]

let me know if you need help

[dengemann]

@agramfort I will grab some files tomorrow to see what's the fastest way to get started. Adding full support for Meta File Format (mff) might take some time. It seems .raw is nothing but a simple binary including the time courses for each channel. In that case it's not clear to me how the info would be accessed.

[dengemann]

@agramfort and @t3on good news. EGI netstation can just be read in via export to EDF+. Works flawlessly. I'm closing this for now.

[mshamalainen]

Except that the Netstation software sometimes crashes on EDF export. EGI should have fixed this by now.

On Apr 7, e2014, at 9:37, "Denis A. Engemann" notifications@github.com wrote:

@agramfort and @t3on good news. EGI netstation can just be read in via export to EDF+. Works flawlessly. I'm closing this for now.

— Reply to this email directly or view it on GitHub.

The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

[dengemann]

@mshamalainen thanks. Will watch out for that. The next question now is how to concatenate raw / epochs objects alongside the channels dimension. I think we had some issues + more or less advanced discussion about that with @christianbrodbeck.

[christianbrodbeck]

That was https://github.com/mne-tools/mne-python/issues/1029. I had to pause this while working on other stuff, this is how far I got: https://gist.github.com/christianbrodbeck/406e280058573df3c110

[dengemann]

@christianbrodbeck cool. I might take a look at that to make a PR. Would that be fine with you?

[christianbrodbeck]

of course @dengemann it would be great to make progress on it

[mluessi]

I just learned that there is a EGI Matlab toolbox to read EGI MFF files. The EGI toolbox is also used in fieldtrip to read MFF files. So if someone wants to work on this, it would probably be easiest to get the Matlab code and translate it.

[dengemann]

@mluessi it turned out that the EDF+ support with Netstation in not ideal. Parsing the events is a pain and is basically broken. I started translating from Matlab and will open a WIP soon. Looks pretty straight forward so far.

We should start thinking about EGI FIFF constants ;-) cc @mshamalainen @LauriParkkonen

[LauriParkkonen]

I think there are two other options besides EDF+.

1. EGI has its own export-binary format which, AFAIK, is documented. @mluessi, is this what you're now trying to read?
2. EGI is moving / has moved to a new native format (combination of XML and binary) and they indicated that they would distribute libraries for reading/writing that format. Unfortunately, I don't know whether the move has really been made.

@mluessihttps://github.com/mluessi it turned out that the EDF+ support with Netstation in not ideal. Parsing the events is a pain and is basically broken. I started translating from Matlab and will open a WIP soon. Looks pretty straight forward so far.

We should start thinking about EGI FIFF constants ;-) cc @mshamalainenhttps://github.com/mshamalainen @LauriParkkonenhttps://github.com/LauriParkkonen

—

[dengemann]

I think there are two other options besides EDF+. 1. EGI has its own export-binary format which, AFAIK, is documented. @mluessi, is this what you're now trying to read?

@LauriParkkonen this is what I started implementing, it's called simple binary format (with '.raw' extension). See this PR #1218.

1. EGI is moving / has moved to a new native format (combination of XML and binary) and they indicated that they would distribute libraries for reading/writing that format. Unfortunately, I don't know whether the move has really been made.

I think fieldtrip includes a Java binary that does this. Does not sound like an option for us. The simple binary seems to be sufficient and more complete than the EDF+ export.

[mluessi]

@LauriParkkonen for 2) (the MFF file format) there is a Matlab toolbbox and EGI is currently working with the fieldtrip guys on implementing support for reading it. It could well be that the Java binary that @dengemann mentioned is the "Matlab toolbox". Nevertheless, I think EGI would probably be willing to provide documentation to someone implementing a native Python reader, I can contact them if you want.

[LauriParkkonen]

I think the ability to read the native EGI format (MFF) would be great as then users do not have to export/convert anything and we would have the full set of metainfo. But I agree that this is not a priority. Re triggers: At NeuroSpin, the MEG triggers are mirrored on the EGI system during acquisition. Thus, they could serve for verifying the sync but should not be inserted to the MEG STI channels. I would vote for creating a new STI channel for the EGI triggers when merging the recordings.

11.4.2014 14.45 kirjoitti Martin Luessi notifications@github.com:

@LauriParkkonenhttps://github.com/LauriParkkonen for 2) (the MFF file format) there is a Matlab toolbbox and EGI is currently working with the fieldtrip guys on implementing support for reading it. It could well be that the Java binary that @dengemannhttps://github.com/dengemann mentioned is the "Matlab toolbox". Nevertheless, I think EGI would probably be willing to provide documentation to someone implementing a native Python reader, I can contact them if you want.

— Reply to this email directly or view it on GitHubhttps://github.com/mne-tools/mne-python/issues/1210#issuecomment-40195102.

[Mensen]

Although the topic is closed, I just wanted to ask whether there was any solution to importing mff files or whether the recommended procedure would be to export into the .raw format from Netstation and then import that?

We currently store everything in .mff format, so if I could avoid the long conversion step that would be great as I'm dealing with all night sleep eeg from 256 channels and conversion takes up to 2 hours.

Thanks!

[dengemann]

Hi @Mensen, currently there is no support for the native netstation binaries in MNE-Python. I also won't have time to work on this in the next weeks. Do you by chance know a description of the format, some code to look at? As always, contributions are welcome :-)

[Mensen]

@dengemann, I was hoping to try and write something for python but I'm working on other things at the moment. I've created a repo for the matlab scripts we use to import mff files (which do not use java objects as the ones described above do), which I'll share with you as soon as I get permission from another author.

They use fairly simple matlab language to access the format, but the binary file for the data is not created in a such a straightforward way. Nevertheless I think the translation into python wouldn't be that difficult and would probably help a lot of people with importing from EGI (a very common format), considering there is a movement away from .raw files.

I'll let you know when I can share those matlab mff files.

[dengemann]

@Mensen I agree it's definitely planned for the future. Having a matlab example would be neat. But no pressure, I won't have time to look into this in the next time.

[jona-sassenhagen]

@Mensen did you get around to releasing MATLAB (or other) code to read in MFF files somewhere? I have a bunch of MFF files here I'd like to read into MNE.

[Mensen]

@jona-sassenhagen I gavemy code to read MFF files into Matlab to the guy working directly with MNE a while ago. Not sure what happened in the end. The repo I have is still in a private organisation for the department but if the code to use MFF in matlab is useful I can pass it to you directly.

[agramfort]

please share the code.

if the code is compatible with the BSD license of MNE you might get some help :)

[dengemann]

Yaeah I am that guy. I still have the code but we never did something with it as there was no urgency for our team. I am basically offline these days, if you need extended input or more see you next year folks!

On 28 Dec 2015, at 14:51, Alexandre Gramfort notifications@github.com wrote:

please share the code.

if the code is compatible with the BSD license of MNE you might get some help :)

— Reply to this email directly or view it on GitHub.

[Mensen]

@dengemann Didn't mean "the guy" to sound so harsh ;)

Most of the time I just save the EEG files as .raw from EGI since those are simple binary files that are easy to load. I understand some of the advantages of MFF, but given the current headache they are to convert, they are overall a bigger problem than solution.

[jona-sassenhagen]

FWIW we have advanced EEGLAB support now, so you can export the MFF files to EEGLAB .set format (with Dr. Chennu's plugin) and import that into MNE.

[Mensen]

@jona-sassenhagen That's really helpful to know. Do you know if you can export back to EEGLAB format? I have a bunch of more advanced analysis scripts that run in Matlab and if I could just do the preprocessing in MNE and continue my analysis with Matlab I would probably be more inclined to use for every day analysis of my data.

This should probably be a different thread at this point, but since I have it hijacked, how well does Python + MNE handle huge EEG files from all night sleep recordings? The raw files are 6-12GB depending on how long people have slept. In Matlab I find to pre-process this sort of data I need at least 3x the RAM since it copies the data within a subfunction, and has to hold the original, the copy, and the preprocessed version (e,g, filtered), all in memory at the same time.

[jasmainak]

In Python, you can load data on-demand. Take a look at our option called preload: http://martinos.org/mne/stable/manual/memory.html. It might be helpful for some (but not all) processing steps ...

[larsoner]

You currently have to have the data in memory to do filtering, though. But it shouldn't cause a 3x memory jump. We do hold the data in float64 (double) format, though, so if it's natively single (32 bit) then there will be a 2x memory requirement.

[ramonapariciog]

Hello!, I will put this here https://github.com/ramonapariciog/mne-mff-reader @dengemann @deep-introspection @SherazKhan

[dengemann]

Don't you want to dare making a pull request here?

On Mon, Feb 6, 2017 at 11:59 PM ramonapariciog notifications@github.com wrote:

Hello!, I will put this here https://github.com/ramonapariciog/mne-mff-reader @dengemann https://github.com/dengemann @deep-introspection https://github.com/deep-introspection @SherazKhan https://github.com/SherazKhan

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mne-tools/mne-python/issues/1210#issuecomment-277842473, or mute the thread https://github.com/notifications/unsubscribe-auth/AB0fikA_0QbAu65RqfFcK0_tBNa3gGZfks5rZ6XtgaJpZM4BwDH- .

[larsoner]

Reopening. There has also been discussion in #1687 about this. It's also related to #3953.

@ramonapariciog please open a PR if you have time. If you get stuck we can help you with comments or PRs into your branch. If you don't think you'll have time, let me know and I can try adapting your code.

[ramonapariciog]

Sorry @dengemann & @Eric89GXL, i will open it, I have the time, just give me this saturday that comes to do it. Greetings.

[larsoner]

ping @ramonapariciog. Will you have time for this anytime soon?

[agramfort]

this is done now