search

open-ephys / open-ephys-python-tools

Python code for interacting with the Open Ephys GUI
MIT License
29 stars 18 forks source link

Request: Add way to only load a subset of channels and range of data #12

Open akesner1 opened 2 years ago

akesner1 commented 2 years ago

Sorry for the post if I am missing something. I am wondering if this functionality could be added so we don't have to load an entire recording.

jsiegle commented 2 years ago

For the Binary and HDF5 formats (NWB and KWIK), the data is memory mapped and only loaded upon request. So this is already implemented for those formats.

For the Open Ephys format, the data is all loaded at once, so this is a feature we'll want to add. We don't have a timeline for this yet, but we'll leave this issue open because it's important to do.

JonathanAMichaels commented 2 years ago

I've also just realized that Open Ephys format files are loaded all at once. This is a huge issue, as all my recordings (not Neuropixels, but still large) are in that format and therefore cannot be loaded on most machines.

jsiegle commented 2 years ago

@medengineer we should add this feature ASAP.

Regarding the implementation, I think we should follow the conventions used by SpikeInterface:

For a Recording object, querying the .continuous property will still return all of the data as it does currently, but we will add a get_traces() method which can take start_frame, end_frame, and channels arguments, e.g.:

traces = recording.get_traces(start_frame=0, end_frame=30000, channels=np.arange(10))

would load the first 30k samples for the first 10 channels into memory.

JonathanAMichaels commented 2 years ago

Thanks @jsiegle. I'm not proud of it, but I've worked around this in the meantime by loading the data in matlab (while keeping things int16 as much as possible), then dumping to a binary.

jsiegle commented 2 years ago

This feature is implemented for the Open Ephys format in the latest version (now available to install via PyPI). See the open_ephys.analysis README file for details.

The Open Ephys format loader is only compatible with data saved by v0.6.x of the GUI...let us know if you need to load data saved by an earlier version, and we can port the code to the archive branch.