Too many spikes detected

[WaveformDai]

Hi, KiloSort team and users of KiloSort,

I'm currently using KS to do spike sorting on linear array but the result is quite confusing:

1) There are two many spikes detected(Figure 1). Current experiment is less than 1000 secs in duration and we have 24 electrodes in total, which may result in 300000 spikes or so at most. But the output spike numbers of KS is at the level of several millions. Figure 1

2) We can not get reasonable waveforms(Figure 2). I think this may be related to the first problem. Figure 2

I try to change ops.Nfilt(from 24 to 72) and ops.maxFR(from 2000 to 20000)， but the problems still exist. It's really a great appreciation if anyone can give me any hint or help me solve these confusing problems.

FYI, I didn't recompute principle component(namely,I didn't change ops.wPCA) and I'm sure our data is of good quality(Figure 3 shows data from one channel). Figure 3

[nsteinme]

Hi, I think it is most likely that the data are not being loaded properly into kilosort, perhaps because ops.NchanTOT does not match the number of rows that are really in the data file. First try loading a bit of the data directly from the file yourself:

fid = fopen('myfile', 'r'); mydata = fread(fid, [ops.NChanTOT 10000], '*int16'); fclose(fid)

Plot one of the rows of mydata. Does it look correct, like what you showed in your other screenshot? If you got it wrong, it will look like garbage, maybe easier to see even with an image of the data (imagesc(mydata)).

If that looks good, try stopping kilosort after the whitening stage and do the same thing for the temp.wh file that it creates (now it will have ops.Nchan rows, rather than ops.NchanTOT).

On Mon, Oct 9, 2017 at 4:42 AM, WaveformDai notifications@github.com wrote:

Hi, KiloSort team and users of KiloSort,

I'm currently using KS to do spike sorting on linear array but the result is quite confusing:

1.

There are two many spikes detected(Figure 1). Current experiment is less than 1000 secs in duration and we have 24 electrodes in total, which may result in 300000 spikes or so at most. But the output spike numbers of KS is at the level of several millions. Figure 1 [image: toomanyspikes] https://user-images.githubusercontent.com/14290870/31324584-b8b20b94-ace6-11e7-8bfe-db0a87e864e1.png 2.

We can not get reasonable waveforms(Figure 2). I think this may be related to the first problem. Figure 2 [image: nogoodwaveform] https://user-images.githubusercontent.com/14290870/31324585-be50682a-ace6-11e7-9815-0357f8866078.png

I try to change ops.Nfilt(from 24 to 72) and ops.maxFR(from 2000 to 20000)， but the problems still exist. It's really a great appreciation if anyone can give me any hint or help me solve these confusing problems.

FYI, I didn't recompute principle component(namely,I didn't change ops.wPCA) and I'm sure our data is of good quality(Figure 3 shows data from one channel). Figure 3 [image: trialdata] https://user-images.githubusercontent.com/14290870/31324592-d5783a8c-ace6-11e7-8a30-d9498c2cdb70.jpg

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cortex-lab/KiloSort/issues/90, or mute the thread https://github.com/notifications/unsubscribe-auth/AHPUPywvOK2XuHO6JaBDlMZXJ-Ktai-Uks5sqZY-gaJpZM4Px-mo .

[WaveformDai]

Hi, Nick, @nsteinme I've checked what you suggested and I'm quite sure that data loading is correct both before and after whitening stage.

Everything is fine until running into mexMPmuFEAT in fullNsxMPMU.m. When running on simulated data, the output spike number of mexMPmuFEAT is about several hundred in a batch, but the spike number increases to tens of thousands in a batch when running on my own data. I'd like to know have you ever met this kind of problem and which parameters do you think may cause so huge a deviation?

[marius10p]

Can you check what the templates look like? Press o in Phy to switch between templates and snippets. The matrix DATA also holds some of the whitened data. You can double-checking it looks normal.

[WaveformDai]

The attached figures are 1) example data after high-pass filtering and spatial whitening;

2) number of spikes in each batch of all channels;

3) templates after sorting;

4) example template in phy;

I use two different datasets from different electrodes(linear array and utah array) and the program still output unreasonable number of spikes and the snippets are still in a mess.

Many thanks for your patience and suggestions!

[WaveformDai]

Good news! It seems that I've found where the problem appears.

In the original code, DATA have an overlap of length ops.ntbuff between nearby batches(there is an overlap of length 2*ops.ntbuff between the 1st and the 2nd batch) and it's subtracted when calculating spike times in fullMPMU.m. But when I load my own data, there is no such overlap(I noticed the overlap at the beginning but I didn't know it will cause so much confusion) and that's the reason that causes messy snippets and unreasonable spike numbers.

Again thank you for developing such efficient tools and kindly answering my doubts.

Lastly, I'd like to know if there is anyone who is using this tool in Beijing and if possible, we could meet and discuss how to use this tool more efficiently and any other things about electrophysiological experiments.

[marius10p]

Glad to hear it worked out, and thanks for your patience. There is indeed a buffer on each batch. If I hear of anyone using Kilosort in Beijing, I'll let you know. I'll send out a tweet, since we don't have a mailing list (yet).

[lauritk]

Hi, how did you actually solve this? We also have unrealistic high spike counts in the clusters. Many of them have almost 2 million spikes.

Edit: Got it sorted, but still not sure what helpt.

[Qifan94]

Hi, I met the same problem. How did you solve this at last?