Decompress Zarr to Binary Files

[atombysx]

Hi all, this is particularly about spikeinterface backward-compatibility with SpikeGLX/catGT: I'm currently compressing two NPX2.0 probes recordings with WavPack (Zarr) and save as Zarr and it seems very good in terms of compression (~60-70% reduced). But one concern is how easy it is to convert back to spikeGLX binary files as we have NIDQ events recorded and need to synchronise the two probes or something we miss before compression and we can revert back to do catGT preprocessing. Any chance spikeinterface has an easy way to deal with this?

[h-mayorquin]

Hi, as far as I know, there is no way in Spikeinterface to accomplish this. I would suggest that the better way around this would be to implement the routine of catGT on spikeinterface. Altought, there must be a good reason why we don't have something like this yet. Any ideas @alejoe91 ?

[alejoe91]

We do have a catgt like processing available. This is essentially a highoass filter + phase shift + CMR. You van check out a tutorial here: https://spikeinterface.readthedocs.io/en/stable/how_to/analyze_neuropixels.html

For synchronization, you should read and store the nidq events separately from the recording. Note that if you use the 385th channel, you can also load that into spikeinterface with the load_sync_channel=True option in the read_spikeglx function

[h-mayorquin]

Oh, I stand corrected. I could not find a reference to catGT in the preprocessing section which is where I expected:

https://spikeinterface.readthedocs.io/en/stable/how_to/analyze_neuropixels.html#preprocess-the-recording

Two questions and I guess we can close this issue: 1) Should we mention in the analysis of neuropixels how to that that preprocessing is like catGT? 2) Do you think it would be useful for users to have a preprocessing step call catGt that just applies highpass filter + phase shift + CMR as a pipe?

[alejoe91]

hey @h-mayorquin

1. Yes we should definitely mention that this is similar to CatGT!
2. I don't think we should. I think our API is clearer and also instructive :) Having a tutorial that shows the chain and explains the different steps is good enough for me (but we should make it more discoverable)

[h-mayorquin]

Makes sense, another reason against 2 is that it will tie us to a level of support that we might now want as we have discussed previously with quality metric naming. I can do the brief correction on the how-to when I have some time.

[atombysx]

Hi all, thanks for the answers. I can do most of the things with spikeinterface and I guess I can just convert the spikeinterface binary file to spikeGLX format if necessary. Is there any documentation of the format of the spikeinterface binary file?

[zm711]

It's just a flat binary file, what specs were you hoping for?

[atombysx]

It's just a flat binary file, what specs were you hoping for?

Well you can have different dtypes, headers and flat raw binary files are 8 bits, especially that the document says spikeinterface binary format is for internal use so it could potentially have extra info stored. Anyway, I went through the source code and found both Spikeinterface and SpikeGLX use int16 on default and there's no extra info - in case anyone wants to know. Thanks everyone for answering patiently! Closing now.

[h-mayorquin]

Thanks a lot for writing what you found. The code for this can be found under write_binary_* on our codebase if anyone is looking.

[samuelgarcia]

1. Yes we should definitely mention that this is similar to CatGT!

We do have a clear ref to catGt here https://spikeinterface.readthedocs.io/en/stable/modules/preprocessing.html#how-to-implement-ibl-destriping-or-spikeglx-catgt-in-spikeinterface