exporting to phy for manual curation

[kshtjkumar]

Hi,

I’ve run spike sorting for several files and have the corresponding waveforms stored. Below is the code I used to display the folder contents:

import os
folder = r"C:\Users\garim\waveforms_kshtj_96lko_23_feb_tetrode_day_0_240223_161515311079"
print(os.listdir(folder))
waveform_folder = os.path.join(folder, 'waveforms')
print(os.listdir(waveform_folder))

this is the output:

['params.json', 'principal_components', 'quality_metrics', 'recording.json', 'recording_info', 'sorting.pickle', 'spike_amplitudes', 'templates_average.npy', 'templates_percentile_1.npy', 'templates_percentile_25.npy', 'templates_percentile_75.npy', 'templates_percentile_98.npy', 'waveforms']
['sampled_index_1.npy', 'sampled_index_2.npy', 'sampled_index_3.npy', 'sampled_index_4.npy', 'waveforms_1.npy', 'waveforms_2.npy', 'waveforms_3.npy', 'waveforms_4.npy']

I have several such folders, and I’d like to perform manual curation since the waveforms and SNR are good, but the ISI ratio is high. I’m still new to manual curation, so any resources or guidance would be appreciated.

Is it possible to load the existing results and convert them to export_to_phy ? I’d prefer not to rerun the spike sorting for each file, as it’s time-consuming. What would be the best way to approach this?

Thanks!

[zm711]

Hey @kshtjkumar,

which version of spikeinterface are you using? You should be to reload your data either as a waveform extractor (version < 0.101.0) or sorting analyzer (version >= 0.101.0) and then use the export_to_phy. we also offer manual curation through the spikeinterface-gui as well as through sortingview.

[kshtjkumar]

i am currently using 0.100.6 version, can you help on how to load it in phy or spikeinterface GUI ?

[zm711]

Could you post your script for analyzing your data? Basically you just need your waveform extractor and that can be used as an input to export_to_phy. If you don't already have spikeinterface-gui you might need @samuelgarcia to comment on the version you'll need since the current version of the gui only works with version > 0.101.0.

[kshtjkumar]

I used mountainsort 5 to sort the data using scheme 2. The waveform folder of each recording has these file :

['params.json', 'principal_components', 'quality_metrics', 'recording.json', 'recording_info', 'sorting.pickle', 'spike_amplitudes', 'templates_average.npy', 'templates_percentile_1.npy', 'templates_percentile_25.npy', 'templates_percentile_75.npy', 'templates_percentile_98.npy', 'waveforms']
['sampled_index_1.npy', 'sampled_index_2.npy', 'sampled_index_3.npy', 'sampled_index_4.npy', 'waveforms_1.npy', 'waveforms_2.npy', 'waveforms_3.npy', 'waveforms_4.npy']

[samuelgarcia]

For spikeinterface-gui you need a sorting analyzer. You have some very basic example how to use it here on the readme: https://github.com/SpikeInterface/spikeinterface-gui

The easiest way is to use spikeinterface 0.101.0 and this

from spikeinterface.widgets import plot_sorting_summary
plot_sorting_summary(sorting_analyzer, curation=True, backend="spikeinterface_gui")

[kshtjkumar]

@zm711 ?

[kshtjkumar]

Hi, so i created a new environment installing 0.101.0. Now i tried the codes on this: https://github.com/SpikeInterface/spikeinterface-gui but it requires reading and sorting from scratch, I don't want to re sort my recordings because it takes a lot of time.

is it possible to do any of this based on the pre sorted recordings ? i have a folder called waveforms from each recording generated after sorting and the files in it are the same that i mentioned above.

Could you please give a clarity on if the manual sorting is possible based on these files or do i resort every file again ! @samuelgarcia @zm711

[zm711]

Well there are two components to this. You would never need to "resort" between version. If you are okay with the sorting results we have various read_x functions and those all work between versions. For example if you sorted with any version of kilosort or did phy stuff we have both read_kilosort and read_phy functions that will work for you :)

I think what you worried about is if the waveform extractor data will still be useful and it is. We have a backward compatibility function just for this case. Do you want to read the docs on it and then if you have more questions let us know!

https://spikeinterface.readthedocs.io/en/stable/tutorials/waveform_extractor_to_sorting_analyzer.html

Once you've converted your waveform extractor into a sorting analyzer you'll be able to use the spikeinterface-gui :)

[kshtjkumar]

hi @zm711, this was really helpful, i used spikeinterface 0.100.6 and ran the following, it worked perfectly:

waveform_folder_path = r"C:\Users\garim\waveforms_kshtj_961_aug_2023_lko_230801_151701347713"
new_sorting_analyzer_path = r"C:\Users\garim\testing_phy"
wvf_extractor = si.load_waveforms(folder=waveform_folder_path)
the_recording = wvf_extractor.recording
the_sorting = wvf_extractor.sorting
channel_locations =wvf_extractor.get_channel_locations()

from spikeinterface.exporters import export_to_phy
we = si.extract_waveforms(recording=the_recording, sorting=the_sorting, folder=waveform_folder_path+str('xyz'))
_  = computespike_amplitudes(waveform_extractor=we)
_  = compute_principal_components(waveform_extractor=we, n_components=3, mode='by_channel_global')
export_to_phy(waveform_extractor=we, output_folder=new_sorting_analyzer_path+str("abc"))

this opens perfectly in phy (but ofc not in spikeinterface gui!) and now am having trouble with understanding phy.

[kshtjkumar]

@zm711 so the previous code is giving trouble now, same version , did anything change ? the recording.json is present and the same lines executed earlier but now it gives this error:

[alejoe91]

What was the previous version?

[kshtjkumar]

for all the analysis the version used is/was 0.100.6 @alejoe91

[kshtjkumar]

@alejoe91 i tried reinstalling but same error persists! Please suggest how to fix this ?

[kshtjkumar]

@zm711 hi, sorry to bother again, but could you please help here! A lot is stuck due to this.

[zm711]

Sorry @kshtjkumar I had a presentation at work so was off of SI/github issues. I need to get re-oriented. So you basically did your analysis with 100.6 so with a waveform extractor. could you show us the contents of your waveform folder/ share your json files in that folder. did you move anything (for example from a server to local or vice versa?). Best case scenario you share the json here. Second best would be you share edited for privacy json files.

[kshtjkumar]

waveforms_kshtj_961_aug_2023_lko_230801_151701347713.zip here is the zipped folder @zm711 ! hope this helps.

[zm711]

So this says the file was originally on your E drive, but you moved it to your c drive for this. inside the recording.json it says relative_paths=False which means that the folders can't be moved around for the WaveformExtractor I think we had a work around right @alejoe91 something like use_relative_paths or use_relative_path when you make the WaveformExtractor. Could you try doing this analysis on the E drive with your recording in the same place as is indicated in the recording.json. We've made improvements to these mechanisms with the SortingAnalyzer so our rec would still be to update and use the backward compatibility code, but for your current stuff it might be that absolute locations are being used.

[kshtjkumar]

hi @zm711 this works perfectly! I shifted my waveform folder to E drive and everthing gets loaded! once i mannually group some clusters i save the phy file and then relaod it in the psikeinterface to generate waveform graphs, I am using this code:

from spikeinterface.extractors import read_phy
phy_sorting = read_phy(r"C:\Users\garim\testing_ptttthyabc_here_i_am") #saving the phy file in C drive

colors = ['Olive', 'Teal', 'Fuchsia']
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
for i, unit_id in enumerate(phy_sorting.unit_ids[1:3]):
    wf = we.get_waveforms(unit_id)
    color = colors[i]
    ax.plot(wf[:, :, 0].T, color=color, lw=0.3)

I hope this approach is correct !