Add support for the NWB format

NeuralEnsemble / python-neo

Neo is a package for representing electrophysiology data in Python, together with support for reading a wide range of neurophysiology file formats

http://neo.readthedocs.io/en/latest/

BSD 3-Clause "New" or "Revised" License

322 stars 247 forks source link

Add support for the NWB format #221

Closed apdavison closed 3 years ago

apdavison commented 8 years ago

NWB = "Neurodata Without Borders" https://neurodatawithoutborders.github.io

specification - https://github.com/NeurodataWithoutBorders/specification
Python APIs - (1) https://github.com/AllenInstitute/nwb-api/tree/master/ainwb (2) https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/core/nwb_data_set.py (3) https://github.com/NeurodataWithoutBorders/api-python
sample datasets @ CRCNS - https://crcns.org/NWB
sample datasets @ Allen Institute - http://alleninstitute.github.io/AllenSDK/cell_types.html#neurodata-without-borders

neuromusic commented 8 years ago

wvangeit commented 8 years ago

neuromusic commented 7 years ago

as a heads-up @apdavison et al, NWB is going through a major refactor, including a new read/write API: https://github.com/NeurodataWithoutBorders/pynwb

@ajtritt & @oruebel are leading up the effort.

the coming months might be a ripe time to get NWB support implemented in Neo

apdavison commented 7 years ago

Thanks, @neuromusic, I had gathered something was going on from some of your recent tweets. Is the specification itself being revised much, or just the Python API? Where should I look to follow what's going on?

neuromusic commented 7 years ago

everything should settle down by ~ SfN. there's a lot of work being done right now, including on documentation and continuous integration

importantly, the NWB stack has been split up into an API layer, a schema specification layer, and a backend layer.

pynwb is the new API

nwb-schema specifies the schema in a bunch of yml files

There are some backward-incompatible changes to the schema (hence the 2.0 designation) but it's not an overhaul. Maybe we can try to get the NEO schema to become NWB 3.0 😄

There's a google group setup, but there's nothing there beyond the announcement of the overhaul. You'l probably get more out of watching the issues on those two repos.

apdavison commented 7 years ago

Great, thanks for the links!

rgerkin commented 5 years ago

@apdavison @neuromusic Has anyone taken this up? This seems pretty timely. I could try to get a student on it if no one else has made any progress.

oruebel commented 5 years ago

@rgerkin with NWB:N 2.0 now releases, I agree that this is a good time for this. I am not aware of any specific efforts right now with regard to integration NWB with neo. If you need help from the NWB development team then this may also be a good project to consider for the next NWB hackathon in May.

bendichter commented 5 years ago

@rgerkin yes this is the perfect time for integration between NWB and neo! I agree with @oruebel that this would be a great project for the next developer hackathon May 15 and 16 at Janelia. Let us know if you are interested in sending people and we'll reserve space for them. If attending that hackathon is not possible, or if you want to get started beforehand, I'm happy to help.

A great way to reach out to the NWB community about these types of things is the nwb-users slack channel. I'd be happy to send invites if you provide emails.

JuliaSprenger commented 5 years ago

@legouee: Didn't you start looking into this at one point?

legouee commented 5 years ago

Yes, effectively, I started working on it, in order to add support to neo rawio. It is currently under progress. Recently I tried to read a real Allen Institute file. I still have to fix some problems. I have not tested yet the NWB:N 2.0. Is there a deadline for this neo support to be added ?

JuliaSprenger commented 5 years ago

I think it would be good to include this in the 0.8 release. However there is no set timeline for this release. @apdavison, @samuelgarcia How about aiming for early March?

apdavison commented 5 years ago

Many thanks @rgerkin, @oruebel and @bendichter for the offers of help and advice.

Given the complexity of NWB, I think early March is probably too ambitious, I'd rather aim for 0.9. It would be nice to get a first version into master by the end of April, and then maybe have a project at the Janelia hackathon to polish/improve/stress-test the implementation?

I'll discuss this with @legouee to figure out how best to proceed.

oruebel commented 5 years ago

@apdavison that timeline and approach sounds reasonable to me. BTW, here the link to the NWB Hackathon website. Registration is not open yet, but we have on the website a link to the form to request an invitation to the hackathon.

rgerkin commented 4 years ago

Much progress in this fork by @legouee: Tasks remaining include:

[ ] Handle SpikeTrain, ImageSequence, etc. (i.e. not just AnalogSignal, Block, etc.)
[ ] Write analyses performed on read data to analysis section of file (i.e. not just writing data back to acquisition section)
[ ] Load and preserve content from disk that can't be read into Neo, so it can be written back into an NWB file intact (i.e. no roundtrip data loss for Neo-incompatible fields in NWB file).

apdavison commented 4 years ago

@rgerkin @bendichter @oruebel I've made a PR (#796) for a first usable version of NWB support in Neo (based on @legouee's fork). It's rough around the edges, much work still to do, but I think it's ready for people to try.

JuliaSprenger commented 3 years ago

A first implementation is merged via #796