:warning: Please note, LINDI is currently under development and should not yet be used in practice.
HDF5 as Zarr as JSON for NWB
LINDI provides a JSON representation of NWB (Neurodata Without Borders) data where the large data chunks are stored separately from the main metadata. This enables efficient storage, composition, and sharing of NWB files on cloud systems such as DANDI without duplicating the large data blobs.
LINDI provides:
This project was inspired by kerchunk and hdmf-zarr and depends on zarr, h5py and numcodecs.
pip install lindi
Or from source
cd lindi
pip install -e .
import pynwb
import lindi
# URL of the remote NWB file
h5_url = "https://api.dandiarchive.org/api/assets/11f512ba-5bcf-4230-a8cb-dc8d36db38cb/download/"
# Set up a local cache
local_cache = lindi.LocalCache(cache_dir='lindi_cache')
# Create the h5py-like client
client = lindi.LindiH5pyFile.from_hdf5_file(h5_url, local_cache=local_cache)
# Open using pynwb
with pynwb.NWBHDF5IO(file=client, mode="r") as io:
nwbfile = io.read()
print(nwbfile)
# The downloaded data will be cached locally, so subsequent reads will be faster
import json
import lindi
# URL of the remote NWB file
h5_url = "https://api.dandiarchive.org/api/assets/11f512ba-5bcf-4230-a8cb-dc8d36db38cb/download/"
# Create the h5py-like client
client = lindi.LindiH5pyFile.from_hdf5_file(h5_url)
client.write_lindi_file('example.lindi.json')
# See the next example for how to read this file
import pynwb
import lindi
# URL of the remote .nwb.lindi.json file
url = 'https://lindi.neurosift.org/dandi/dandisets/000939/assets/56d875d6-a705-48d3-944c-53394a389c85/nwb.lindi.json'
# Load the h5py-like client
client = lindi.LindiH5pyFile.from_lindi_file(url)
# Open using pynwb
with pynwb.NWBHDF5IO(file=client, mode="r") as io:
nwbfile = io.read()
print(nwbfile)
import json
import lindi
# URL of the remote .nwb.lindi.json file
url = 'https://lindi.neurosift.org/dandi/dandisets/000939/assets/56d875d6-a705-48d3-944c-53394a389c85/nwb.lindi.json'
# Load the h5py-like client for the reference file system
# in read-write mode
client = lindi.LindiH5pyFile.from_lindi_file(url, mode="r+")
# Edit an attribute
client.attrs['new_attribute'] = 'new_value'
# Save the changes to a new .nwb.lindi.json file
client.write_lindi_file('new.nwb.lindi.json')
import lindi
# URL of the remote .nwb.lindi.json file
url = 'https://lindi.neurosift.org/dandi/dandisets/000939/assets/56d875d6-a705-48d3-944c-53394a389c85/nwb.lindi.json'
# Load the h5py-like client for the reference file system
# in read-write mode with a staging area
with lindi.StagingArea.create(base_dir='lindi_staging') as staging_area:
client = lindi.LindiH5pyFile.from_lindi_file(
url,
mode="r+",
staging_area=staging_area
)
# add datasets to client using pynwb or other tools
# upload the changes to the remote .nwb.lindi.json file
See this example.
Special Zarr annotations used by LINDI
See LICENSE.