We should provide clean and reliable waveforms as part of our standard spike sorting output.
This task will reuse the code from Chris to extract the waveforms and depend on the spike sorting.
To make sure the extraction doesn't take too long:
the waveform extractor should be written for high performance, here are a few pointers to improve:
we need to pre-compute the channel correspondancy map and feed it to the function. This is trivial, but we should re-use the dart code to make sure we are fully consistent with them.
the default working dimension is [nc, ns] so that time samples are contiguous in memory
there should be 2 options: 1) return single waveforms, 2) return sum of waveforms and count. The implementation of 2) will use scipy.sparse and aggregate the results using subindices of the original map. This will save a lot of memory and be much faster.
this code will be parrellized, with workers actuating on chunks of time of the raw data, writing distinct buckets of waveforms for each worker
the pre-processing should be simplified: we'll swap the k-filter that creates artefacts at the summed waveform level (butterflies) and we'd like to do a simple CAR instead, which will also be faster
what is the strategy for the number of aggregated waveforms (median probably) ? Should we stop at a given number to try and have a fixed signal to noise ratio (like stack order = 500) ? Should we read the whole file ?
If the stack order is limited, we could write a fancy algorithm that optimizes raw data I/O: what is the set of raw ephys time chunks that achieve a given stack order with the minimum raw electrophysiology length ?
[x] define a format and dataset type
[ ] write the waveform extractor for high performance
We should provide clean and reliable waveforms as part of our standard spike sorting output. This task will reuse the code from Chris to extract the waveforms and depend on the spike sorting.
To make sure the extraction doesn't take too long:
the waveform extractor should be written for high performance, here are a few pointers to improve:
dartcode to make sure we are fully consistent with them.scipy.sparseand aggregate the results using subindices of the original map. This will save a lot of memory and be much faster.this code will be parrellized, with workers actuating on chunks of time of the raw data, writing distinct buckets of waveforms for each worker
the pre-processing should be simplified: we'll swap the k-filter that creates artefacts at the summed waveform level (butterflies) and we'd like to do a simple CAR instead, which will also be faster
what is the strategy for the number of aggregated waveforms (median probably) ? Should we stop at a given number to try and have a fixed signal to noise ratio (like stack order = 500) ? Should we read the whole file ? If the stack order is limited, we could write a fancy algorithm that optimizes raw data I/O: what is the set of raw ephys time chunks that achieve a given stack order with the minimum raw electrophysiology length ?
[x] define a format and dataset type
[ ] write the waveform extractor for high performance
[ ] create task and write unit test
[ ] add to the ephys pipeline
[ ] update integration test
[ ] run on SDSC for all past ephys insertions
[ ] register datasets