Request : Add compatibility with next gen BRW files

[b-grimaud]

Hi,

3Brain is currently moving towards BRW v4.x, compatible with newer versions of their software but most importantly adapted to multiwell chips, which means that recordings from multiple parallel chips can be stored in the same file. This implies a major rework of the underlying HDF5 architecture, and the current BiocamIO reader is not compatible.

I got the OK from 3Brain to share their documentation on that file format, but it would still probably be better to not have it publically hosted.

Is there someone I can reach out to that could take a look at that documentation and possibly figure out how to update the reader ?

Thanks !

[JuliaSprenger]

Hi @BptGrm, thanks for reaching out. This already the right place for this discussion. Do you also already have a dataset in the new format? With a tiny test dataset to be added to GIN we could start extending the current BiocamIO.

[JuliaSprenger]

@alejoe91 Do you have an idea on how to extend this?

[alejoe91]

Hi @JuliaSprenger

Unfortunately not. I think that one will need to explore the new h5 test files and see where they save the relevant stuff. It would be great to ask support from someone from 3Brain directly.

[JuliaSprenger]

@BptGrm Any news on this from your side? As mentioned above, feel free to contribute a small test dataset to GIN to start extending the neo IO.

[b-grimaud]

Hi, thanks for bringing this up again !

I submitted a very basic PR to the HerdingSpikes spike sorter to read newer BrainWave files. This might be a starting point but it's definetly not compatible with the multi-well recording that this format is meant to support.

I dug pretty deep in the file structure at the time but there were still some aspects that I couldn't quite figure out, namely how they manage data chunks.

I tried reaching out to 3Brain about this issue a while ago, but I didn't get any answers. As I mentionned in the original post of this issue, they did send me some documentation on the format that I could send through email.

I have a few recordings in this format that I could share, I'll get started on the GIN upload procedure soon.

[JuliaSprenger]

Hi @BptGrm Thanks for the update. This sounds like it will still take some time and a lot of tinkering to figure out the new format without someone from 3Brain helping out. Unless you require neo supporting the new format I would say we postpone this until 3Brain contributes or someone knowing the format in more detail has a chance to look at it.

[b-grimaud]

I'll ask 3Brain again and see if I can get a more positive answer.

In the meantime, if anyone desperately need to open newer BrainWave file through neo, there's another open source project called Xenon LFP analysis which contains a downsampling tool that can convert those files to the older format. Setting the 'new' sampling rate to the original should do the trick.

[Lychfindel]

Hi all, I am also very interested for the support of brw 4.x. I could also access the official documentation of the format, which are very extensive and include also some code snippet to extract the information in python. How is proceeding the implementation? I'm willing to help and contribute if there is the need.

[b-grimaud]

@Lychfindel As of #1326 it is now supported.

[alejoe91]

@Lychfindel you need to install NEO from sources!

[Lychfindel]

Thank you! This is a great news! Unfortunately the import of my dataset is still raising errors. After investigating a little bit I understood that the error comes from the type of raw data saved: while the new implementation handles the .brw file with data type "Raw" (i.e. with no compression applied), it does not handle the other two supported raw data types "WaveletBasedEncodedRaw" and "EventsBasedSparseRaw", which involve some type of data compression. Do you think it would be possible to include also these raw types even if they are not completely raw? It should be possible to reconstruct the raw signal (with some data loss) even from these types

[b-grimaud]

Looking through the docs again, those formats are quite a bit more complicated. 3Brain does provide some code examples for both types, but I don't know how much work it would be to transpose them into NEO. The PDF is available on the NeuralEnsemble GIN.

[alejoe91]

In principle it should be possible to load the other compressed formats in NEO.

• For the EventsBasedSparseRaw, this is already supported as one can load spike times and waveforms. I don't think that NEO should fill in the data with zeros, since this is basically making up data. If you use spikeinterface, we can discuss adding padding options there (we might have some already).

• For WaveletBasedEncodedRaw, it could be possible to implement something in NEO, but I honestly would tend towards not supporting this, because it might be hard to maintain. @Lychfindel as a general practice, I recommend storing raw data and not doing lossy compression, and maybe do lossless compression later (e.g., with spikeinterface - see paper)

[Lychfindel]

I was not aware that was possible to load spike times and waveforms, this will definitely work for the EventsBasedSparseRaw. Thanks for the hint!

I agree with you that is always better storing raw data, unfortunately I'm working with a dataset recorded in the past by a different research group. In any case, I completely understand your point to support only raw formats!

[b-grimaud]

Coming back to this, EventsBasedSparseRaw seems quite a bit different from spikes and waveforms. The compressed .brw files only contains sections of raw data, unlike .bxr files, at least as far as I've seen. I'm not exactly sure how BrainWave selects which section to save or not, but that compression mode is available in the basic version of the software, which doesn't include live spike/LFP detection. It might be possible to combine both spikes and raw signal in a single file using the upgraded versions of BrainWave.

I have a working version of the BiocamRawIO module that can read this format, essentially adapted from 3Brain's own documentation. It is pretty slow, mostly because the nested file structure requires parsing bytes by bytes iteratively.

I can open a "work in progress" PR if anyone is interested.

WaveletBasedEncodedRaw seems easier on paper, but would require adding pywt in the biocam optional dependencies.

[mahlzahn]

I only now saw this ongoing discussion and already implemented the reading of EventsBasedSparseRaw in the open PR #1446

It tightly follows the code example given in the 3Brain documentation. We could also make some of the options available as parameters to the init function of the BiocamRawIO class.