Closed behinger closed 2 weeks ago
Looks really nice.
Would you be able to share some of these files? The CNT reader is a quite recent addition and I remember there being some variation in the file format. It would be an interesting project to try and see if we can make it work without the dependencies to libeep.
I'm not sure if you mean script files or .cnt files. You can use the read_antcnt file if you want. The compiled version of libeep I don't really know and you would have to contact ANT / Robert Smies (the current author of libeep).
If you want a .cnt file & trigger files you can find examples here. These are only bipolar luminance sensor channels, I can try to find a small .cnt EEG file if you need it.
Yeah, I meant the data files, thanks. These should be fine. Maybe I'll have some time next week to see if I can make any sense of it.
Might be worth asking the author of the library for source plus license permission
Hi,
Maintainer of libeep here. Our project is located on sourceforge: https://sourceforge.net/projects/libeep/ We use the LGPL license which means you can include a copy of the code in your project, provided you submit back changes you make and keep a reference to the original code.
That said, go ahead and include our project into MNE. If it's possible in git I would recommend pulling the sources directly from our subversion repo at sourceforge to prevent having old copies when we make updates.
I have not looked in detail in your project yet, so I can't comment on the feasibility of it, but I recommend to not use binaries, but instead compile from source each time a user installs your project. From my own experience I know it can be quite a handful to support binaries on multiple platforms(Windows, Linux, etc).
If you need any information on how to build the sources(simple python-dev suite + cmake should suffice), let me know. You can contact me on my github account or at: rsmies(at)ant-neuro(dot)com
Yours, Robert
thanks Robert
mne-python is meant to be a pure python package to work out of the box on all systems.
to support these files we need to rewrite the readers in pure python
I wonder if file specs for these ANT files are available somewhere. @smeeze @behinger ?
We have a document that describes our riff format. You can find it here: http://eeprobe.ant-neuro.com/doc/cnt_riff.txt
We have made same modifications recently that have not been updated in this document yet. Mainly the adjustment to use 64-bit chunk sizes to allow very large files to be written. Files with these offsets can be recognized with the Riff tag 'RF64' instead of 'RIFF'. This deviates from the riff standard though.
apart from using riff as container format, we also use a variant of huffman coding to compress our sample data. This is described in the document above under the 'data' chunk. There is more to our file format, such as legacy file support, but I doubt you will come across such old files.
Hope this helps.
Great, thanks. Let's see if I can make this work.
Has there been any progress on this? I have a .cnt file that fails on load with read_raw_cnt
and I am guessing it is because they are from an ANT Neuro system.
no progress AFAIK. We need volunteers to look into it. @jaeilepp is not working with us anymore.
Ok, then I guess I can volunteer.
Great
Hi all,
I'm really sorry to open up this issue again after one year, but I'm still struggling with importing ANT Neuro .cnt files with MNE. I started a conversation with @behinger and Robert a long time ago, but I didn't manage to solve the issue. In my specific case, I might need to compile the libeep library on my own since I'm using MacOS, but I was wondering whether the problem has been solved in any other way?
Many thanks.
we don't still yet such a reader. Someone has to take a stab at it.
in the meantime you can convert your files to edf or .set and then use mne?
Since you are on macOS you might want to try the Python bindings of Biosig. I don't know if it supports ANT CNT, but it's worth a try.
Hi,
@agramfort : oh, that's a pity :( I think that @behinger created one, but his solution was specifically designed for ubuntu 64bit if remember correctly. Btw, yes, that's what I'm doing at the moment: using .set to work with MNE. I think I'll also check whether our ANT recording software is able to export the raw files directly into edf or some other mne-supported format.
@cbrnr : thank you, I'll take a look at them and see whether they can help me sort the issue out
Many thanks to both of you!
@agramfort and I looked into possible solutions to this issue.
At the moment, the fix involves compiling the libeep library linked above yourself (or using binaries such as those provided by @behinger) and using the Python bindings by copying files to you site-packages. Unfortunately, this solution is Python 2.7 only. Moving to Python 3 would involve providing our own bindings for this library and this is not a clean solution.
We concluded that a better option and long term solution is to contact ANT and see if they can provide support for their file formats.
ANT recording system(eego) could export edf, .eeg(brainvision) or .cnt(neuroscan).
Hello, are there any advantages to the ANT neuro .cnt format over the neuroscan .cnt format? As mentioned above our ANT system has the ability to export to .cnt(neuroscan) so I'm wondering what would be the motivation to ever export to ANT's format. Is there increased precision?
I have no idea
I emailed someone from ANT Neuro who responded that
It mostly just has a higher bit-depth and more detailed information for events (triggers that have names). With lower bit depth file formats, you may need to apply a high-pass filter first in order to reduce DC offsets.
So it seems the data does not exactly match between ANT Neuro .cnt format and Neuroscan .cnt format
I have no objection to add a reader for ANT data but it just needs to be implemented by someone. Maybe suggest this to ANT engineer? we have received help from different companies in the past to support their file format.
Since our group work extensively with both the ANT-Neuro system and MNE-Python, I just want to chime in here with some updates and tentative paths forward.
Status quo: We still don't have a native reader of ANT-Neuro EEProbe Continuous Data (.cnt) files in MNE-Python. We cannot use the mne.io.read_raw_cnt() method that imports Neuroscan .cnt files for this purpose. They are different file formats that just unfortunately happen to coincide on picking the same .cnt extension.
Tentative solutions:
mne.io.BaseRaw
. I'm personally going with option 1 simply because I do some stuff in MATLAB too. But all are equally viable. Note that the BrainVision files exported by ANT-Neuro all have a missing space between 'Brain' and 'Vision', so you would need to either fix that before reading or adjust the byte length in mne.io.read_raw_brainvision (found by @proloyd).
Next step: The obstacle right now is that most of the source code in libeep is written in C. To conform to MNE-Python's goal of being a standalone Python package, someone needs to completely rewrite these C functions in Python instead of simply adding a Python wrapper. This will likely take a while, and not sure if anyone from ANT-Neuro is going to do it. When I find the time I might just take a stab at this myself, but very unlikely until late 2023.
Next step: The obstacle right now is that most of the source code in libeep is written in C. To conform to MNE-Python's goal of being a standalone Python package, someone needs to completely rewrite these C functions in Python instead of simply adding a Python wrapper. This will likely take a while, and not sure if anyone from ANT-Neuro is going to do it. When I find the time I might just take a stab at this myself, but very unlikely until late 2023.
And as another blocker the license on SourceForge is listed as GPLv2, which is incompatible with MNE-Python's BSD clause. So these functions cannot be used as a reference / looked at while writing equivalent Python code. To be able to do that, you'd have to contact the original authors and get permission to relicense/adapt their code as BSD in MNE-Python.
Download the source code of the data I/O library behind the EEGLAB importer (called libeep) here and use the provided python wrapper (import libeep) to read such EEProbe .cnt files. One would then manually create the MNE-Python raw object by grabbing all relevant information from the "cnt" object instance created by libeep.
One option would be to improve the libeep
Python bindings so that it import libeep
could have a libeep.read_raw_ant
or whatever that returned a libeep.RawANT
instance that subclasses mne.io.BaseRaw
. The BaseRaw
interface has been established for long enough now that I think as long as the _read_segment_file
is implemented properly and __init__
creates info
properly, it wouldn't be too fragile. The tough part would be creating pip- and conda-installable libraries, but nowadays with cibuildwheel
and conda-forge
it's not insurmountable.
@larsoner great suggestion... We'd also need the license permission from ANT-Neuro to update their Python bindings of libeep
, is that right?
You only need to relicense if you want to include code in MNE-Python. My "One option" / second paragraph above if you license your libeep
Python library as GPLv2 as well everything should be okay. No new code is added to MNE-Python so BSD only matters in your pylibeep
(or whatever you call it) in that you would subclass BaseRaw (which should be fine -- GPL code can use BSD but not vice-versa).
@mh105 I would consider contacting ANT support to ask them for a Python reader. We managed to get this from EGI/Philips who released mffpy. But it took sometime to get this...
@larsoner I realized what @behinger provided is exactly doing the subclassing mne.io.BaseRaw
route. I've updated my comment above so that people know that option 2 above
Download the source code of the data I/O library behind the EEGLAB importer (called libeep) here and use the provided python wrapper (import libeep) to read such EEProbe .cnt files...
is already pretty much set. I agree @agramfort that we should just have a pip installable package from ANT instead of managing the c library ourselves... I'll give it a try to persuade ANT, might cc you guys on the email.
I met the same problem, @agramfort Is there any progress in the contacting ANT support to Python
a customer of ANT should reach out to them. I don't have any contact with them
I contacted the support this morning for a python reader. We are using a lot this system in Geneva, let's wait for their reply.
From what I heard is that ANT Neuro is working on a pip installable package, and it will likely be released later this year.
I had a chat with Eshwar Ghumare, product manager @ ANT. They have an updated version of the SDK which will come out early next year (march / april) with alpha versions by the end of this year. It will include a python interface (he mentionned a pure python implementation, but I have some doubts based on the interaction with the SDK he mentionned). He'll keep me in the loop and will send my way the alpha/beta versions for testing.
Overall, I'm +1 to remove this issue from the sprint-2023
and have it tackled through a distributed python package in 2024.
any update on this from Eshwar Ghumare, @mscheltienne? I skimmed the ANT-Neuro website and don't see anything about Python in their downloads area (where their MATLAB stuff is).
No news.. not on this or on 2 other matters transfered to him by ANT's support.
A bit of an update as I had to play with cnt files this week, on Windows and Linux. I have little hope that ANT will provide a pure python reader or will handle the maintenance of a pip and conda installable libeep
package. The last update was 4 years ago. That said:
One option would be to improve the libeep Python bindings so that it import libeep could have a libeep.read_raw_ant or whatever that returned a libeep.RawANT instance that subclasses mne.io.BaseRaw. The BaseRaw interface has been established for long enough now that I think as long as the _read_segment_file is implemented properly and init creates info properly, it wouldn't be too fragile. The tough part would be creating pip- and conda-installable libraries, but nowadays with cibuildwheel and conda-forge it's not insurmountable.
I never packaged python library which require C extension and which need to be build differently on all platforms, and I'd like to learn. I started playing @ https://github.com/mscheltienne/antio where I have 3 branches:
cmake
libeep
existThe CI build is split in 2 parts:
(1) Build the binaries required by the module libeep
-> works on all platforms.
(2) Copy those binaries to the module libeep
and build wheels with cibuildwheel
.
And that's where I'm stuck at the moment. Even building locally, the wheel generated with cibuildwheel
are broken and are not installable (I actually got a wheel for antio
which depends on... antio
).
On the other hand, running:
pip install build
python -m build
Does create working wheels.
I'll keep looking into it, but if anyone has the time to look at it and point me in the right direction, I'd love to learn the best packaging practice here 😉
I think this looks good, but it's difficult to debug, because the actions fail in the testing step.
That said, have you looked at build tools that might make this task a bit easier out of the box? I'm thinking of scikit-build-core, which does have native cmake support and also includes an example.
I think this looks good, but it's difficult to debug, because the actions fail in the testing step.
Locally, disabling testing, the generated wheels can't even be installed. Purely broken..
I'm thinking of scikit-build-core, which does have native cmake support and also includes an example.
I'll have a look, I'm really looking for what might be the best practice here.
Probably compared to my previous message, I was thinking that I will move the libeep
branch as a folder within the main
branch, and have this folder included in the sdist
through MANIFEST.in
. Then the cmake
commands would be invoked in setup.py
, this way the sdist
would actually include everything needed to build the package.
Probably compared to my previous message, I was thinking that I will move the
libeep
branch as a folder within themain
branch, and have this folder included in thesdist
throughMANIFEST.in
. Then thecmake
commands would be invoked insetup.py
, this way thesdist
would actually include everything needed to build the package.
Yes, it is certainly better to have everything in one place. However, I'd still take a look at scikit-build-core, which should make it easier to build (no messing around with MANIFEST.in
etc).
Update of the day:
src/antio
containing the python code and src/libeep
containing the dependency, in the main
branch.cmake
files, and my limited experience in code compiling makes it difficult. The less I change libeep
, the better.src/antio
and src/libeep
, inspired from the documentation of scikit-build-core led me to include everything you need in the sdist
and a scripting build in setup.py
which invokes cmake
and moves the built files to the correct location before building the wheel. The sdist
now includes everything you need and will work as long as you have cmake
installed.python -m build
works and give me working wheels on all ubuntu-latest
, windows-latest
, macos-13
(intel) and macos-14
(arm), on python 3.9 to 3.12. c.f. https://github.com/mscheltienne/antio/actions/runs/10353511600cibuildwheel
is evading me. I spent hours trying to figure out how to get it to work on the manylinux
docker image locally, still no luck.@mscheltienne I could look into the cibuildwheel to see if I have any quick ideas to try. Should I just open a PR to https://github.com/mscheltienne/antio if I have anything useful?
Sure! :pray:
If you find anything, can you please ping me here (or in the relevant PR)? I've been struggling with cibuildwheel in the past as well, so I'm curious what the problem is.
For now, what I have are a mix of cibuildwheel
and setuptools
issues:
python3-devel
with yum
because the cmake
commands needs it to run.quay.io/pypa/manylinux2014_x86_64
runs old versions (as expected), and I can't get setuptools
to a high enough version to use pyproject.toml
entirely, except if I install a more recent python version through what I figure is the equivalent of a ppa
on Ubuntu.setuptools
(59.6.0) behaves differently from the recent one, the variable self.build_lib
does not contain the path to the python package being built as in the recent version. Overall, the way I run the extension build with build_ext
in setup.py
is flawed.(1) makes sense because you probably need Python.h
. Re (2), why do you have to use such an old version? Can you not install e.g. 2_28 with pipx (https://cibuildwheel.pypa.io/en/stable/setup/)? This would also solve (3), and I really think it doesn't make sense to build with such ancient versions.
I don't think you should need to do (1) or (2) (and hence not deal with (3)) manually, cibuildwheel
should do this for you. But yes if you use the docker image directly yourself rather than invoking python -m cibuildwheel
locally you'll need to replicate however they manage to install Python in cibuildwheel
.
... in other words, I'm testing by doing:
git clone ... && cd antpy
CIBW_BUILD=cp312-* python -m cibuildwheel --archs native
I removed the stuff about before-all
and I get to:
[ 90%] Built target EepObjects
[ 95%] Linking C static library libEepStatic.a
[ 95%] Built target EepStatic
[100%] Linking C shared library libEep.so
[100%] Built target Eep
error: [Errno 2] No such file or directory: '/tmp/tmp3ljxt5i0/python/pyeep.so'
which seems like progress?
/tmp/tmp3ljxt5i0/python/pyeep.so
I was at this point and figured out the update of python because of this. Anyway, I will now dissect what you did to get it working, thanks a lot @larsoner!
Great job @mscheltienne for making the python wrapper working! Our lab stumbled upon this a longtime ago, and we had to use their matlab importer as a work around. However, their libeep package does not expose a lot of header information, i.e., patient id, date of birth, machine info, channel types, start time etc etc. I am working on including that via libeep, so that those fields can be populated in the rawANT object. I will make a pull request shortly with my changes.
Also, I would like to add the impedance values to the rawANT.info, since it is something we look at in the lab. Any idea how we shall approach this? @larsoner I remember mne.Raw object supporting only single set of values for them.
ANT Neuro has a special fileformat (.cnt and .trg files). I could not find any support to import them in MNE. They do have a c-library libeep which you can compile for python. They were so nice to send me a precompiled 64bit linux version (thus I did not really try to compile myself, but should be straight forward) and I did an early sketch of an importer script for MNE-python. You can find this script here. For me this script seems to work nicely. So far without any preload capacities, but using libeep should be straight forward.
As I don't have much experience in mne-python I did not really know any guides to follow. I adapted a mix between the read_eeglab and the read_cnt functions.
If you do use the libeep-tool be sure to include a link to the source somewhere.
Best, Benedikt