Run comparison between different sorted outputs (automated and manual)

[mvdm]

We want to be able to compare the output of different automated sorters, among themselves and with "ground truth" manual datasets.

Some possibilities:

• SpikeInterface has a module: https://spikeinterface.readthedocs.io/en/latest/sorters_comparison.html
• Ephys-Viz: https://github.com/flatironinstitute/ephys-viz
• MountainView: https://github.com/flatironinstitute/qt-mountainview
• SpikeForest: https://github.com/flatironinstitute/spikeforest2

[mvdm]

Using SpikeInterface may be the fastest way to get multiple sorted outputs we can visualize and compare. But will still need a way to load MClust output too.

[manimoh]

Initial attempt in this notebook. As things stand now, spike comparison found no matches between any of the Kilosort2 iteration and the mclust output :disappointed: Not even a single unit matched and the ‘agreement scores’ are all really low.

[manimoh]

Fixing time alignment of Mclust data has improved the agreement scores (See updated notebook.)

[mvdm]

Next step: visualizing this in PHY (seems to work best in Windows..) and comparing with Jimmie's manually sorted waveforms. Can run Create_CQ_Files.m to generate *wv files containing average waveforms. This needs the .ntt files to run, so may need to ask Jimmie to do.

[mvdm]

Discussed some interesting next steps with Yarik today:

• [ ] Export kilosort outputs to NWB file, so that it can be loaded in MATLAB (and others) as an input to Jimmie's data analysis pipeline. Would require writing some sort of wrapper to that the NWB unit loader creates a ts object in the workspace (like LoadSpikes([])). Seems like spikeinterfaces can already save to NWB (@manimoh)
• [ ] Once we have ability to load kilosort output and plug it in to existing data analysis pipelines (@jgmaz), we can test how metrics of interest like decoding accuracy and GLM variance explained depend on kilosort parameters, and how they compare to manual sorting. Hypothesis: automated sorting will outperform manual sorting for ensemble decoding as we allow more units, but then GLM variance explained will go down.