Merging Clusters causes loss of valid ISIs data

[juliencarponcy]

Hello,

I have observed that ALL the ISIs < spkRefrac_merge_ms are automatically deleted when you are merging two clusters. Thus this is likely to cause the (permanent) loss of valid ISIs data <2ms (default parameter), which then, cause a very dodgy side effect: You are very unlikely to have violation of your refractory period if you are deleting all ISIs < 2ms, and thus, a lot of users could consider clusters as "single units" just because they have merged two clusters together. Could you please investigate this further ? Just to confirm or try to find out what I got wrong ? (I have to perform all of my sorting again because of that. I'll let you know if I notice other side effects)

Thank you very much for your beautiful and convenient work anyway.

Best,

Julien

[jamesjun]

Thanks for raising a valid point. I think I chose a too high refractory period to merge double-detection of action potential events. I often encounter double detection of a single AP event in dense electrode array and I merge them based on the waveform similarity while allowing time shift. However, the time separation of a double detection rarely exceeds 0.5 ms. The default value is 2 ms and I agree that it's too high. I will reduce this parameter to 0.5 msec and see how it affects the spike sorting accuracy on our ground truth dataset.

-James

On Fri, Nov 15, 2019 at 7:42 AM juliencarponcy notifications@github.com wrote:

Hello,

I have observed that ALL the ISIs < spkRefrac_merge_ms are automatically deleted when you are merging two clusters. Thus this is likely to cause the (permanent) loss of valid ISIs data <2ms (default parameter), which then, cause a very dodgy side effect: You are very unlikely to have violation of your refractory period if you are deleting all ISIs < 2ms, and thus, a lot of users could consider clusters as "single units" just because they have merged two clusters together. Could you please investigate this further ? Just to confirm or try to find out what I got wrong ? (I have to perform all of my sorting again because of that. I'll let you know if I notice other side effects)

Thank you very much for your beautiful and convenient work anyway.

Best,

Julien

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/ironclust/issues/34?email_source=notifications&email_token=ACEHBOB5NMRVSS3776QPG63QT2KMRA5CNFSM4JN2JPQ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HZTGQFQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACEHBOAD5F4VVENHBKTTPB3QT2KMRANCNFSM4JN2JPQQ .

[juliencarponcy]

Thanks for the quick reply,

Anyway, I think that whatever the parameter you can end up with some minor issues... I'll follow up later on that and provide you a few examples. Two different problematic cases for me is, what happen when a small waveform is detected over a bigger spike, and how to deal with the merging of clusters which have been detected on two different sites. (Very dependant on the lag you can have between the two peaks). Thus indeed you don't want spkRefrac_merge_ms to be too small neither.. As I said I'll provide a few examples later.

Thanks very much,

Julien

[kouichi-c-nakamura]

Thanks James.

what happen when a small waveform is detected over a bigger spike, and how to deal with the merging of clusters which have been detected on two different sites.

Julien and I are literally talking about these issues! I could not see an easy solution.

[juliencarponcy]

Hi James,

I guess there is two sort of problems that we do not really have clue on how to resolve them.

The first one is when two different spikes are detected close to one another (Trace view and most left clusters on Cluster view). The problem arise in the fact that when you are merging clusters then, I feel that it gave rise to genuine merging + spike detected twice, and I don't know very much how to control for that.

The other occurs when the same spike is detected on (very) distant contacts (Black and Red clusters on cluster view). In that case also, I have observed genuine merging + double detection. For that I guess that the best chance is to play on maxDist_site_um AND/OR maxDist_site_spk_um. Is that possible to specify the first one larger that the second one, and how much you think increasing too much this parameter could impede detection of other genuine spikes occuring very close in time and space ?

In other words, it is very tricky for us to be sure of our combination between spkRefrac_ms , spkRefrac_merge_ms,maxDist_site_um , and maxDist_site_spk_um. I would be much more keen to redo my sorting if I could be more certain of the outcome of the combination of these parameters, but that's obviously a lot of combination to test and results for us would just be empirical.

Any thoughts on that ?

Thank you very much for your help,

Julien

PS : I would be happy to share a dataset if you think that could help you in having a better idea