read_phy(): excluding units marked as noise

[dejanica93]

Hello,

I have been using mountainsort5 for spike sorting and phy to curate and inspect the units. After I have further curated the units in phy, I wanted to get the unit IDs, excluding those marked as "noise" using read_phy() :

sorting_curated_phy = se.read_phy(sorting_output_path, exclude_cluster_groups=["noise"])

However, I get wrong unit IDs. I think as I manually curate the units, the IDs change, but it does not seem to get updated when I try to load using the read_phy() function. Do you know why that is or how I could fix this?

If I load spikes directly from files "spike_clusters.npy" and "spike_times.npy", the unit IDs are correct (the same as I see them in phy), but I can't exclude the units marked as "noise" in that case.

The phy version I am using is: phy TemplateGUI v2.0b5 and phylib v2.4.3.

Thank you for your time!

Best, Dejana

[zm711]

Howdy @dejanica93 (Dejana), couple of follow up questions. Are you mostly merging or splitting in Phy? The way phy handles merging and splitting requires giving new "spikeinterface" ids so that could be contributing. You could try:

sorting_curated_phy.get_property('original_cluster_id') and see if that returns the ids you were expecting? If those IDs are the same then the different numbers you are seeing are because spikeinterface is keeping track of its own system rather than the "Phy" numbering system. There should be a file called cluster_group.tsv. You could also look in that file and see the number systems and see if they align with what you are expecting.

Let us know what you see!

[alejoe91]

Thanks @zm711

Hi Dejana! I think this issue is exactly what Zach mentioned. Basically SpikeInterface IDs can be also strings, so we internally keep track of the SI unit ids (which are different than the Phy unit ids).

[dejanica93]

Thank you for responding so quickly, @zm711 and @alejoe91 !

I tried with sorting_curated_phy.get_property('original_cluster_id') and got None. But I loaded the file called cluster_group.tsv and there the unit IDs are "updated" after merging and splitting (I do both, but probably splitting more), and I can also see which are marked as noise. Yaay!

This helped a lot! Thank you both!

[alejoe91]

Dejana, can you try to install from source? We might have recently added this property (I'll check tomorrow and let you know!)

[dejanica93]

I installed it from source now too in a new environment and tried running again sorting_curated_phy.get_property('original_cluster_id') , but I still get None. I tried sorting_curated_phy.get_property('firing_rates') as well, but that gives the same output.

[alejoe91]

You can check the available properties with: sorting.get_property_keys()

[alejoe91]

Could you share the cluster_info.csv?

[dejanica93]

I see. The available properties seem to be ['amp', 'ch', 'channel_group', 'depth', 'fr', 'quality', 'n_spikes', 'sh'], so that explains the Nones. :D

If I run sorting_curated_phy = se.read_phy(sorting_output_path, exclude_cluster_groups=["noise"]) and sorting_curated_phy.get_unit_ids(), I still get the original unit IDs when installed from source.

I have a cluster_info.tsv file, but I cannot share that file type here so I will e-mail it to you.

[alejoe91]

@dejanica93 I tested on my side and the original_cluster_id is loaded correctly, so I'm not entirely sure what's going wrong in your case. Can you zip and send me the entire phy folder (excluding the dat files)?

[alejoe91]

Hi Dejana,

Everything works well on my side on the dataset you provided:

Can you print: pip list | grep spikeinterface?

[dejanica93]

Sorry for the delay. I am not sure if you wanted me to check the spikeinterface version or something else. I am using spikeinterface-poetry, but this is what I got from pip list:

spikeinterface  0.97.1
spikeinterface-poetry  4.0.0

I did also try to run everything in a new environment when installing from source as you asked earlier, and there it was:

spikeinterface  0.100.0.dev0 C:\Users\Dejana\PycharmProjects\git\spikeinterface

[alejoe91]

yeah, not sure why it's failing. It's working on my side! It could be a Windows-related parsing issue

[zm711]

sorry to be clear @dejanica93, you're saying it worked it 0.100.0.dev right? But that it failed in 0.97.1.

Maybe it was added between. I get it to parse fine on Windows for 100.dev.

[alejoe91]

@zm711 from what I understood it failed on 0.100 too

[zm711]

when installing from source as you asked earlier, and there it was:

Trying to clarify this statement. There it was could be referring to there the original_cluster_id.

[alejoe91]

or there it was could be referred to pip list | grep spikeinterface :)

Let's wait for @dejanica93 input

[dejanica93]

Sorry for the confusion. I meant that original_cluster_id didn't work with either version when I tried it at that time. I tried again now with both versions just to be sure, and now it works, haha! It is also with the updated/new unit ID numbers after merging/splitting spikes. I am not sure what happened in between.

Thank you for your help, and sorry for the mess!

[zm711]

@alejoe91 is right :) !, but happy it is working now.