ENH: import ANT Neuro .CNT files (some code available)

behinger commented 7 years ago

ANT Neuro has a special fileformat (.cnt and .trg files). I could not find any support to import them in MNE. They do have a c-library libeep which you can compile for python. They were so nice to send me a precompiled 64bit linux version (thus I did not really try to compile myself, but should be straight forward) and I did an early sketch of an importer script for MNE-python. You can find this script here. For me this script seems to work nicely. So far without any preload capacities, but using libeep should be straight forward.

As I don't have much experience in mne-python I did not really know any guides to follow. I adapted a mix between the read_eeglab and the read_cnt functions.

If you do use the libeep-tool be sure to include a link to the source somewhere.

Best, Benedikt

jaeilepp commented 7 years ago

Looks really nice.

Would you be able to share some of these files? The CNT reader is a quite recent addition and I remember there being some variation in the file format. It would be an interesting project to try and see if we can make it work without the dependencies to libeep.

behinger commented 7 years ago

I'm not sure if you mean script files or .cnt files. You can use the read_antcnt file if you want. The compiled version of libeep I don't really know and you would have to contact ANT / Robert Smies (the current author of libeep).

If you want a .cnt file & trigger files you can find examples here. These are only bipolar luminance sensor channels, I can try to find a small .cnt EEG file if you need it.

jaeilepp commented 7 years ago

Yeah, I meant the data files, thanks. These should be fine. Maybe I'll have some time next week to see if I can make any sense of it.

larsoner commented 7 years ago

Might be worth asking the author of the library for source plus license permission

ghost commented 7 years ago

Hi,

Maintainer of libeep here. Our project is located on sourceforge: https://sourceforge.net/projects/libeep/ We use the LGPL license which means you can include a copy of the code in your project, provided you submit back changes you make and keep a reference to the original code.

That said, go ahead and include our project into MNE. If it's possible in git I would recommend pulling the sources directly from our subversion repo at sourceforge to prevent having old copies when we make updates.

I have not looked in detail in your project yet, so I can't comment on the feasibility of it, but I recommend to not use binaries, but instead compile from source each time a user installs your project. From my own experience I know it can be quite a handful to support binaries on multiple platforms(Windows, Linux, etc).

If you need any information on how to build the sources(simple python-dev suite + cmake should suffice), let me know. You can contact me on my github account or at: rsmies(at)ant-neuro(dot)com

Yours, Robert

agramfort commented 7 years ago

thanks Robert

mne-python is meant to be a pure python package to work out of the box on all systems.

to support these files we need to rewrite the readers in pure python

jaeilepp commented 7 years ago

I wonder if file specs for these ANT files are available somewhere. @smeeze @behinger ?

ghost commented 7 years ago

We have a document that describes our riff format. You can find it here: http://eeprobe.ant-neuro.com/doc/cnt_riff.txt

We have made same modifications recently that have not been updated in this document yet. Mainly the adjustment to use 64-bit chunk sizes to allow very large files to be written. Files with these offsets can be recognized with the Riff tag 'RF64' instead of 'RIFF'. This deviates from the riff standard though.

apart from using riff as container format, we also use a variant of huffman coding to compress our sample data. This is described in the document above under the 'data' chunk. There is more to our file format, such as legacy file support, but I doubt you will come across such old files.

Hope this helps.

jaeilepp commented 7 years ago

Great, thanks. Let's see if I can make this work.

sdeslauriers commented 6 years ago

Has there been any progress on this? I have a .cnt file that fails on load with read_raw_cnt and I am guessing it is because they are from an ANT Neuro system.

agramfort commented 6 years ago

no progress AFAIK. We need volunteers to look into it. @jaeilepp is not working with us anymore.

sdeslauriers commented 6 years ago

Ok, then I guess I can volunteer.

agramfort commented 6 years ago

Great

langestroop commented 5 years ago

Hi all,

I'm really sorry to open up this issue again after one year, but I'm still struggling with importing ANT Neuro .cnt files with MNE. I started a conversation with @behinger and Robert a long time ago, but I didn't manage to solve the issue. In my specific case, I might need to compile the libeep library on my own since I'm using MacOS, but I was wondering whether the problem has been solved in any other way?

Many thanks.

agramfort commented 5 years ago

we don't still yet such a reader. Someone has to take a stab at it.

in the meantime you can convert your files to edf or .set and then use mne?

cbrnr commented 5 years ago

Since you are on macOS you might want to try the Python bindings of Biosig. I don't know if it supports ANT CNT, but it's worth a try.

langestroop commented 5 years ago

Hi,

@agramfort : oh, that's a pity :( I think that @behinger created one, but his solution was specifically designed for ubuntu 64bit if remember correctly. Btw, yes, that's what I'm doing at the moment: using .set to work with MNE. I think I'll also check whether our ANT recording software is able to export the raw files directly into edf or some other mne-supported format.

@cbrnr : thank you, I'll take a look at them and see whether they can help me sort the issue out

Many thanks to both of you!

sdeslauriers commented 5 years ago

@agramfort and I looked into possible solutions to this issue.

At the moment, the fix involves compiling the libeep library linked above yourself (or using binaries such as those provided by @behinger) and using the Python bindings by copying files to you site-packages. Unfortunately, this solution is Python 2.7 only. Moving to Python 3 would involve providing our own bindings for this library and this is not a clean solution.

We concluded that a better option and long term solution is to contact ANT and see if they can provide support for their file formats.

XiaTaopsycho commented 5 years ago

ANT recording system(eego) could export edf, .eeg(brainvision) or .cnt(neuroscan).

pstetz commented 3 years ago

Hello, are there any advantages to the ANT neuro .cnt format over the neuroscan .cnt format? As mentioned above our ANT system has the ability to export to .cnt(neuroscan) so I'm wondering what would be the motivation to ever export to ANT's format. Is there increased precision?

agramfort commented 3 years ago

I have no idea

pstetz commented 3 years ago

I emailed someone from ANT Neuro who responded that

It mostly just has a higher bit-depth and more detailed information for events (triggers that have names). With lower bit depth file formats, you may need to apply a high-pass filter first in order to reduce DC offsets.

So it seems the data does not exactly match between ANT Neuro .cnt format and Neuroscan .cnt format

agramfort commented 3 years ago

I have no objection to add a reader for ANT data but it just needs to be implemented by someone. Maybe suggest this to ANT engineer? we have received help from different companies in the past to support their file format.

mh105 commented 1 year ago

Since our group work extensively with both the ANT-Neuro system and MNE-Python, I just want to chime in here with some updates and tentative paths forward.

Status quo: We still don't have a native reader of ANT-Neuro EEProbe Continuous Data (.cnt) files in MNE-Python. We cannot use the mne.io.read_raw_cnt() method that imports Neuroscan .cnt files for this purpose. They are different file formats that just unfortunately happen to coincide on picking the same .cnt extension.

Tentative solutions:

Import EEProbe .cnt files in MATLAB with the EEGLAB importer provided by ANT-Neuro here and save as .set files. Then use mne.io.read_raw_eeglab() to read into MNE-Python as a raw object.
Download the source code of the data I/O library behind the EEGLAB importer (called libeep) here and use the provided python wrapper (import libeep) to read such EEProbe .cnt files. One would then manually create the MNE-Python raw object by grabbing all relevant information from the "cnt" object instance created by libeep. This is done in the functions provided by @behinger by subclassingmne.io.BaseRaw.
When exporting recordings from the ANT-Neuro laptop, select a different format (BrainVision, EDF+, Neuroscan) from the EEProbe .cnt file format. You can then use appropriate mne.io.read_raw() functions to read them into MNE-Python. This is acceptable except slight differences on bit depth and loss of certain trigger events. For instance the impedance check events are ignored in these different formats, and you don't get the impedance values at the beginning and end of recording. Amplifier disconnect/reconnect events will also be omitted in these other formats (but EEProbe .cnt file do include them)!

I'm personally going with option 1 simply because I do some stuff in MATLAB too. But all are equally viable. Note that the BrainVision files exported by ANT-Neuro all have a missing space between 'Brain' and 'Vision', so you would need to either fix that before reading or adjust the byte length in mne.io.read_raw_brainvision (found by @proloyd).

Next step: The obstacle right now is that most of the source code in libeep is written in C. To conform to MNE-Python's goal of being a standalone Python package, someone needs to completely rewrite these C functions in Python instead of simply adding a Python wrapper. This will likely take a while, and not sure if anyone from ANT-Neuro is going to do it. When I find the time I might just take a stab at this myself, but very unlikely until late 2023.

larsoner commented 1 year ago

Next step: The obstacle right now is that most of the source code in libeep is written in C. To conform to MNE-Python's goal of being a standalone Python package, someone needs to completely rewrite these C functions in Python instead of simply adding a Python wrapper. This will likely take a while, and not sure if anyone from ANT-Neuro is going to do it. When I find the time I might just take a stab at this myself, but very unlikely until late 2023.

And as another blocker the license on SourceForge is listed as GPLv2, which is incompatible with MNE-Python's BSD clause. So these functions cannot be used as a reference / looked at while writing equivalent Python code. To be able to do that, you'd have to contact the original authors and get permission to relicense/adapt their code as BSD in MNE-Python.

Download the source code of the data I/O library behind the EEGLAB importer (called libeep) here and use the provided python wrapper (import libeep) to read such EEProbe .cnt files. One would then manually create the MNE-Python raw object by grabbing all relevant information from the "cnt" object instance created by libeep.

One option would be to improve the libeep Python bindings so that it import libeep could have a libeep.read_raw_ant or whatever that returned a libeep.RawANT instance that subclasses mne.io.BaseRaw. The BaseRaw interface has been established for long enough now that I think as long as the _read_segment_file is implemented properly and __init__ creates info properly, it wouldn't be too fragile. The tough part would be creating pip- and conda-installable libraries, but nowadays with cibuildwheel and conda-forge it's not insurmountable.

mh105 commented 1 year ago

@larsoner great suggestion... We'd also need the license permission from ANT-Neuro to update their Python bindings of libeep, is that right?

larsoner commented 1 year ago

You only need to relicense if you want to include code in MNE-Python. My "One option" / second paragraph above if you license your libeep Python library as GPLv2 as well everything should be okay. No new code is added to MNE-Python so BSD only matters in your pylibeep (or whatever you call it) in that you would subclass BaseRaw (which should be fine -- GPL code can use BSD but not vice-versa).

agramfort commented 1 year ago

@mh105 I would consider contacting ANT support to ask them for a Python reader. We managed to get this from EGI/Philips who released mffpy. But it took sometime to get this...

mh105 commented 1 year ago

@larsoner I realized what @behinger provided is exactly doing the subclassing mne.io.BaseRaw route. I've updated my comment above so that people know that option 2 above

Download the source code of the data I/O library behind the EEGLAB importer (called libeep) here and use the provided python wrapper (import libeep) to read such EEProbe .cnt files...

is already pretty much set. I agree @agramfort that we should just have a pip installable package from ANT instead of managing the c library ourselves... I'll give it a try to persuade ANT, might cc you guys on the email.

Xiezhibin commented 1 year ago

I met the same problem, @agramfort Is there any progress in the contacting ANT support to Python

agramfort commented 1 year ago

a customer of ANT should reach out to them. I don't have any contact with them

mscheltienne commented 1 year ago

I contacted the support this morning for a python reader. We are using a lot this system in Geneva, let's wait for their reply.

mh105 commented 1 year ago

From what I heard is that ANT Neuro is working on a pip installable package, and it will likely be released later this year.

mscheltienne commented 1 year ago

I had a chat with Eshwar Ghumare, product manager @ ANT. They have an updated version of the SDK which will come out early next year (march / april) with alpha versions by the end of this year. It will include a python interface (he mentionned a pure python implementation, but I have some doubts based on the interaction with the SDK he mentionned). He'll keep me in the loop and will send my way the alpha/beta versions for testing.

Overall, I'm +1 to remove this issue from the sprint-2023 and have it tackled through a distributed python package in 2024.

drammock commented 3 months ago

any update on this from Eshwar Ghumare, @mscheltienne? I skimmed the ANT-Neuro website and don't see anything about Python in their downloads area (where their MATLAB stuff is).

mscheltienne commented 3 months ago

No news.. not on this or on 2 other matters transfered to him by ANT's support.

mscheltienne commented 4 weeks ago

A bit of an update as I had to play with cnt files this week, on Windows and Linux. I have little hope that ANT will provide a pure python reader or will handle the maintenance of a pip and conda installable libeep package. The last update was 4 years ago. That said:

One option would be to improve the libeep Python bindings so that it import libeep could have a libeep.read_raw_ant or whatever that returned a libeep.RawANT instance that subclasses mne.io.BaseRaw. The BaseRaw interface has been established for long enough now that I think as long as the _read_segment_file is implemented properly and init creates info properly, it wouldn't be too fragile. The tough part would be creating pip- and conda-installable libraries, but nowadays with cibuildwheel and conda-forge it's not insurmountable.

I never packaged python library which require C extension and which need to be build differently on all platforms, and I'd like to learn. I started playing @ https://github.com/mscheltienne/antio where I have 3 branches:

libeep_gitlab: a mirror of the gitlab branch
libeep: a step-down simpler version which generates the binaries we need in the python package with cmake
main: a python package within which a module libeep exist

The CI build is split in 2 parts: (1) Build the binaries required by the module libeep -> works on all platforms. (2) Copy those binaries to the module libeep and build wheels with cibuildwheel.

And that's where I'm stuck at the moment. Even building locally, the wheel generated with cibuildwheel are broken and are not installable (I actually got a wheel for antio which depends on... antio). On the other hand, running:

pip install build
python -m build

Does create working wheels.

I'll keep looking into it, but if anyone has the time to look at it and point me in the right direction, I'd love to learn the best packaging practice here 😉

cbrnr commented 3 weeks ago

I think this looks good, but it's difficult to debug, because the actions fail in the testing step.

That said, have you looked at build tools that might make this task a bit easier out of the box? I'm thinking of scikit-build-core, which does have native cmake support and also includes an example.

mscheltienne commented 3 weeks ago

I think this looks good, but it's difficult to debug, because the actions fail in the testing step.

Locally, disabling testing, the generated wheels can't even be installed. Purely broken..

I'm thinking of scikit-build-core, which does have native cmake support and also includes an example.

I'll have a look, I'm really looking for what might be the best practice here.

Probably compared to my previous message, I was thinking that I will move the libeep branch as a folder within the main branch, and have this folder included in the sdist through MANIFEST.in. Then the cmake commands would be invoked in setup.py, this way the sdist would actually include everything needed to build the package.

cbrnr commented 3 weeks ago

Probably compared to my previous message, I was thinking that I will move the libeep branch as a folder within the main branch, and have this folder included in the sdist through MANIFEST.in. Then the cmake commands would be invoked in setup.py, this way the sdist would actually include everything needed to build the package.

Yes, it is certainly better to have everything in one place. However, I'd still take a look at scikit-build-core, which should make it easier to build (no messing around with MANIFEST.in etc).

mscheltienne commented 3 weeks ago

Update of the day:

Rework the structure to have src/antio containing the python code and src/libeep containing the dependency, in the main branch.
scikit-build-core: I gave it a shot, but I'm getting further away from getting a working wheel with it. It looks like I would need to rewrite the cmake files, and my limited experience in code compiling makes it difficult. The less I change libeep, the better.
Still, the structure src/antio and src/libeep, inspired from the documentation of scikit-build-core led me to include everything you need in the sdist and a scripting build in setup.py which invokes cmake and moves the built files to the correct location before building the wheel. The sdist now includes everything you need and will work as long as you have cmake installed.
python -m build works and give me working wheels on all ubuntu-latest, windows-latest, macos-13 (intel) and macos-14 (arm), on python 3.9 to 3.12. c.f. https://github.com/mscheltienne/antio/actions/runs/10353511600
cibuildwheel is evading me. I spent hours trying to figure out how to get it to work on the manylinux docker image locally, still no luck.

larsoner commented 3 weeks ago

@mscheltienne I could look into the cibuildwheel to see if I have any quick ideas to try. Should I just open a PR to https://github.com/mscheltienne/antio if I have anything useful?

mscheltienne commented 3 weeks ago

Sure! :pray:

cbrnr commented 3 weeks ago

If you find anything, can you please ping me here (or in the relevant PR)? I've been struggling with cibuildwheel in the past as well, so I'm curious what the problem is.

mscheltienne commented 3 weeks ago

For now, what I have are a mix of cibuildwheel and setuptools issues:

(1) I need to install python3-devel with yum because the cmake commands needs it to run.
(2) the quay.io/pypa/manylinux2014_x86_64 runs old versions (as expected), and I can't get setuptools to a high enough version to use pyproject.toml entirely, except if I install a more recent python version through what I figure is the equivalent of a ppa on Ubuntu.
(3) This older version of setuptools (59.6.0) behaves differently from the recent one, the variable self.build_lib does not contain the path to the python package being built as in the recent version. Overall, the way I run the extension build with build_ext in setup.py is flawed.

cbrnr commented 3 weeks ago

(1) makes sense because you probably need Python.h. Re (2), why do you have to use such an old version? Can you not install e.g. 2_28 with pipx (https://cibuildwheel.pypa.io/en/stable/setup/)? This would also solve (3), and I really think it doesn't make sense to build with such ancient versions.

larsoner commented 3 weeks ago

I don't think you should need to do (1) or (2) (and hence not deal with (3)) manually, cibuildwheel should do this for you. But yes if you use the docker image directly yourself rather than invoking python -m cibuildwheel locally you'll need to replicate however they manage to install Python in cibuildwheel.

larsoner commented 3 weeks ago

... in other words, I'm testing by doing:

git clone ... && cd antpy
CIBW_BUILD=cp312-* python -m cibuildwheel --archs native

I removed the stuff about before-all and I get to:

[ 90%] Built target EepObjects
[ 95%] Linking C static library libEepStatic.a
[ 95%] Built target EepStatic
[100%] Linking C shared library libEep.so
[100%] Built target Eep
error: [Errno 2] No such file or directory: '/tmp/tmp3ljxt5i0/python/pyeep.so'

which seems like progress?

mscheltienne commented 3 weeks ago

/tmp/tmp3ljxt5i0/python/pyeep.so

I was at this point and figured out the update of python because of this. Anyway, I will now dissect what you did to get it working, thanks a lot @larsoner!

proloyd commented 3 weeks ago

Great job @mscheltienne for making the python wrapper working! Our lab stumbled upon this a longtime ago, and we had to use their matlab importer as a work around. However, their libeep package does not expose a lot of header information, i.e., patient id, date of birth, machine info, channel types, start time etc etc. I am working on including that via libeep, so that those fields can be populated in the rawANT object. I will make a pull request shortly with my changes.

Also, I would like to add the impedance values to the rawANT.info, since it is something we look at in the lab. Any idea how we shall approach this? @larsoner I remember mne.Raw object supporting only single set of values for them.

mne-tools / mne-python

ENH: import ANT Neuro .CNT files (some code available) #3609