search

open-ephys / GUI

Archived source code for the Open Ephys GUI
https://github.com/open-ephys/plugin-GUI
126 stars 282 forks source link

Mixed endianness of the ".continuous" data format #305

Closed galenlynch closed 7 years ago

galenlynch commented 9 years ago

The current format of continuous recording files is very confusing. Although this isn't in the documentation, after looking at the analysis scripts and the RecordingEngine.cpp files themselves, it seems like ".continuous" recording files have little endian headers, followed by a number of records consisting of blocks of little endian timestamps and sample counts and the a block of big endian ints. In effect this makes the continuous recording files a file with alternating big and little endian data, which seems needlessly complicated.

At first I assumed that the big endian data was just the format of the data collected from the Rhythm API, but after looking at the source code for OriginalRecording object that produces ".continuous" files, I see that the big endian recording blocks are big endian seemingly by design: the floating point data from the Rhythm API is converted to big endian integers with a call to AudioDataConverters::convertFloatToInt16BE. For comparison, the HDF5Recording object converts to little endian integers with AudioDataConverters::convertFloatToInt16LE.

What is the reasoning behind this choice? It makes reading data files much more complicated.

Additionally, what is the status of the HDF5Recording object used by the KWIK recorder? Should it be used to make neural recordings instead of the Original recorder?

open-ephys commented 9 years ago

In short, consistency of endianness wasn't really a factor when we were designing the format. It would be nice (and not too difficult) to have an updated version that's consistent across the board, but we've put most of our recording-related development efforts into the KWIK format, which is robust and working well. I'd recommend using that for your recordings; the convenience of loading HDF5 files is amazing. I've been using it for all my data collection lately, and it's been great.

galenlynch commented 9 years ago

Awesome, I'll start using the KWIK format!

nikolaskaralis commented 9 years ago

Is the open-ephys format going to be supported in the future? What about the discussions on the klustaviewas mailing list regarding the format? It seems they might be moving away from the Kwik format.

On Tue, Jul 28, 2015 at 12:17 AM, open-ephys notifications@github.com wrote:

In short, consistency of endianness wasn't really a factor when we were designing the format. It would be nice (and not too difficult) to have an updated version that's consistent across the board, but we've put most of our recording-related development efforts into the KWIK format, which is robust and working well. I'd recommend using that for your recordings; the convenience of loading HDF5 files is amazing. I've been using it for all my data collection lately, and it's been great.

— Reply to this email directly or view it on GitHub https://github.com/open-ephys/GUI/issues/305#issuecomment-125363588.

Nikolas Karalis - http://www.nikolaskaralis.gr http://nikolaskaralis.gr

Neuroscience Graduate Student Ludwig-Maximilians Universität München Centre for Integrative Neuroscience - Universität Tübingen Neuroscience Graduate Université Bordeaux 2 - Charité Universitätsmedizin Berlin Applied Mathematics and Physics Graduate National Technical University of Athens, Greece

open-ephys commented 9 years ago

Yes, we'll continue supporting the original format.

Once the specs for the updated Kwik format are released, we plan to add the ability to record in that format as well. We'll also keep an HDF5-based recording option, which will likely be the Neurodata Without Borders format that's currently under development (http://neurodatawithoutborders.github.io).