search

NeuroML / NeuroML2

This repository hosts the NeuroML 2 Schema, the ComponentType definitions in LEMS and the core documentation of NeuroML2.
https://docs.neuroml.org
GNU Lesser General Public License v3.0
50 stars 24 forks source link

HDF5 in NeuroML2 #93

Open pgleeson opened 8 years ago

pgleeson commented 8 years ago

An HDF5 based equivalent for NeuroML with efficient storage/compression of positions/connections in a network would be a good alternative to serialising in XML. This should be well supported by the various NeuroML APIs so use of the format is transparent for users.

This is being worked on in:

Examples of HDF5 files in this format are here and here.

Some idea of current/future support:

neuroml hdf5 format

ccluri commented 7 years ago

Is there some documentation regarding the HDF5 structure here? Or is this does like xdmf-paraview kind - where the NML2 points to the the HDF5 a dataset array?

Would you consider linking via NSDF which is also HDF5 based?

pgleeson commented 7 years ago

@ccluri The documentation at the moment doesn't go too far beyond the examples above I'm afraid... The priority until now has been getting a format we can read and write in both Python and Java.

What is the current status of NSDF? Are there other tools for it besides the Python API? Are there examples storing network structure?

NSDF compatibility is certainly something to consider for the future, but for now it's best to keep the format as simple as possible, try to keep the apis for it as compatible as possible and see what optimisations are possible. One which is in the python api is allowing libNeuroML to create a numpy array for connectivity data/positions in memory (as opposed to creating a tree of Python objects) and then just putting that as the dataset into the HDF5 file. Makes it much quicker to read/write the info. Is that a scenario you think would be supported by NSDF?