search

SpikeInterface / spikeinterface

A Python-based module for creating flexible and robust spike sorting pipelines.
https://spikeinterface.readthedocs.io
MIT License
533 stars 187 forks source link

Expose `backend_options` at the analyzer level to set `storage_options` and `saving_options` #3446

Closed alejoe91 closed 1 month ago

alejoe91 commented 2 months ago

Generally, default compression works fine, but I've been hitting some errors with chunks being too large when analyzing very long NP recordings (>8h).

This PR gives the possibility to modify the zarr arguments used to create a dataset. These can be passed at the create_sorting_analyzer or save_as step. The update test shows it in action

samuelgarcia commented 2 months ago

Lets have a call for this. I think we should have something more geenric. Could we handle zarr specifics in on dict like the "storage_option" and propagate it at the good places ? I known that the storage option are more the crdential and username but we could use a universal dict for all backend specific stuff like compression also no ? The code here is a bit too zarr specific no ?

alejoe91 commented 2 months ago

@samuelgarcia I refactored it to have a more general backend_kwargs that could support multiple backends (also in the future). Let me know what you think, but I'm sure you'll like it :)