Error in extractTemplatesfromSnippets

[smasri09]

Hello, I am not yet sure if this error is my fault of the software, but I thought I'd post here while I continue to debug it. I'm processing 64ch data recorded from a whitematter system, read into a matlab array and saved an int16 file from matlab. Using the latest version of kilosort.

I'm getting this error:

Error using gpuArray/subsref Subscript indices must either be real positive integers or logicals.

Error in extractTemplatesfromSnippets (line 56) wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));

Error in datashift2 (line 4) [wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

Error in ksGUI/runPreproc (line 726) obj.rez = datashift2(obj.rez, 1);

Error in ksGUI>@(~,~)obj.runPreproc() (line 331) 'Callback', @(~,~)obj.runPreproc());

Error while evaluating UIControl Callback.

On debugging, I'm noticing that 1) dd is a large array of zeros, and 2) this line

% discard empty samples dd = dd(:, 1:k);

changes the dimensions of dd from 61x50000 to 61x0 because k is 0, which I didn't know was possible in matlab, so that's cool.

Also, line 32 returns all zeros, but dataRAW looks real.

[row, col, mu] = isolated_peaks_new(dataRAW, ops);

Any insights would be appreciated.

• Samer Masri

[tchemaly]

I have the same problem.

Error using gpuArray/subsref Subscript indices must either be real positive integers or logicals.

Error in extractTemplatesfromSnippets (line 56) wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));

Error in extract_spikes (line 42) [wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

Error in ksGUI/runSpikesort (line 786) [obj.rez, st3, tF] = extract_spikes(obj.rez);

Error in ksGUI>@(~,~)obj.runSpikesort() (line 337) 'Callback', @(~,~)obj.runSpikesort());

Error while evaluating UIControl Callback.

[gawygawy]

i got the same error too!

Error using gpuArray/subsref Subscript indices must either be real positive integers or logicals.

Error in extractTemplatesfromSnippets (line 56) wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));

Error in extract_spikes (line 42) [wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

[bjslater13]

I am getting the exact same error, just an FYI my data sorted just fine with the older kilosort2. https://github.com/MouseLand/Kilosort/releases/tag/v2.0

I'm using a linear 64ch probe, same settings for both k2 and k2.5/3. I'll look into the differences and see if I can find anything, but it might be worth just using k2 until there's a solution.

[marius10p]

I am not sure these are all the same problem. The error occurs when no spike is found in the raw data. This usually happens due to bad configurations such as wrong numbers of channels, or the data being transposed. Did you all double check in the GUI that the data looks how you'd expect it to? Post some screenshots here please.

@bjslater13, yours looks like a ks3-specific problem, since it worked with ks2. Have you tried ks2.5? That might give us more information.

[smasri09]

Thanks for the comments. It seems that kilosort 2 does fix that error in my case as well. The data was always presented properly in time in the gui, so I don't think it was transposed. I have a new error: error running kilosort! k must be less than or equal to n.

I don't know what k and n are referring to. I do get a 2x2 figure output seemingly showing good sorting statistics, but nothing is saved. This is on a few different files of 64ch and size ranging from 200mb to 3gb. I tried altering ops.reordering and ops.nskip with no luck.

[marius10p]

Kilosort2 does not have that spike detection step. You can try reducing ops.spkTh to -4 or something like that so it can catch more spikes for that step. Are your spikes upside-down by any chance, with the voltage going positive?

To get more informative errors, run Kilosort from a script.

[smasri09]

Well after some digging on errors in ks2 that seemed different, I see it's actually the same problem! This part of my last post. The data is real, and I've been to sort neurons manually, it's also quite noisy. I played with threshold, nskip and ordering variables in ops.

On debugging, I'm noticing that 1) dd is a large array of zeros, and 2) this line

% discard empty samples dd = dd(:, 1:k);

changes the dimensions of dd from 61x50000 to 61x0 because k is 0, which I didn't know was possible in matlab, so that's cool.

[marius10p]

Can you please try my recommendation above and answer my question?

[smasri09]

To use the scripted version? The output is below. My suspicion is that kilosort is having these issues because my recordings have noise with much higher amplitude than the spikes. Could that lead to these issues? This is the error output:

Error using randperm K must be less than or equal to N.

Error in extractTemplatesfromSnippets (line 58) wTEMP = dd(:, randperm(size(dd,2), nPCs));

Error in learnTemplates (line 19) [wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

Error in learnAndSolve8b (line 15) rez = learnTemplates(rez, iorder0);

Error in main_kilosort_edit (line 41) rez = learnAndSolve8b(rez);

And here is console output before the error:

main_kilosort_edit Looking for data inside C:\Users\Samer\Desktop\WhiteMatter\logger_5F1211\runnow\ Time 0s. Determining good channels.. found 121048 threshold crossings in 177.12 seconds of data found 0 bad channels Time 7s. Computing whitening matrix.. Getting channel whitening matrix... Channel-whitening matrix computed. Time 8s. Loading raw data and applying filters... Time 40s. Finished preprocessing 321 batches. random seed for clusterSingleBatches: 1 Obtained 7 PC waveforms in 0.52 seconds time 0.59, pre clustered 1 / 321 batches Batch 13 has fewer than Nfilt spikes. Batch 37 has fewer than Nfilt spikes. Batch 66 has fewer than Nfilt spikes. Batch 67 has fewer than Nfilt spikes. Batch 68 has fewer than Nfilt spikes. Batch 72 has fewer than Nfilt spikes. Batch 73 has fewer than Nfilt spikes. Batch 81 has fewer than Nfilt spikes. Batch 85 has fewer than Nfilt spikes. Batch 86 has fewer than Nfilt spikes. Batch 96 has fewer than Nfilt spikes. Batch 100 has fewer than Nfilt spikes. Batch 101 has fewer than Nfilt spikes. Batch 106 has fewer than Nfilt spikes. Batch 107 has fewer than Nfilt spikes. Batch 112 has fewer than Nfilt spikes. Batch 113 has fewer than Nfilt spikes. Batch 114 has fewer than Nfilt spikes. Batch 115 has fewer than Nfilt spikes. Batch 116 has fewer than Nfilt spikes. Batch 119 has fewer than Nfilt spikes. Batch 120 has fewer than Nfilt spikes. Batch 121 has fewer than Nfilt spikes. Batch 124 has fewer than Nfilt spikes. Batch 198 has fewer than Nfilt spikes. Batch 199 has fewer than Nfilt spikes. Batch 200 has fewer than Nfilt spikes. Batch 201 has fewer than Nfilt spikes. Batch 207 has fewer than Nfilt spikes. Batch 210 has fewer than Nfilt spikes. Batch 297 has fewer than Nfilt spikes. Batch 305 has fewer than Nfilt spikes. Batch 310 has fewer than Nfilt spikes. Batch 311 has fewer than Nfilt spikes. Batch 312 has fewer than Nfilt spikes. Batch 314 has fewer than Nfilt spikes. Batch 315 has fewer than Nfilt spikes. Batch 316 has fewer than Nfilt spikes. Batch 317 has fewer than Nfilt spikes. Batch 318 has fewer than Nfilt spikes. Batch 319 has fewer than Nfilt spikes. Batch 320 has fewer than Nfilt spikes. Batch 321 has fewer than Nfilt spikes. time 0.04, compared 1 / 321 batches time 8.78, Re-ordered 321 batches.

[putnampt]

I am also receiving this error message.

Error using gpuArray/subsref Subscript indices must either be real positive integers or logicals.

Error in extractTemplatesfromSnippets (line 56) wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));

Error in extract_spikes (line 42) [wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

Error in ksGUI/runSpikesort (line 795) [obj.rez, st3, tF] = extract_spikes(obj.rez);

Error in ksGUI>@(~,~)obj.runSpikesort() (line 339) 'Callback', @(~,~)obj.runSpikesort());

Error while evaluating UIControl Callback.

I am working with data from a 16 channel probe that has been written into a binary int16 .dat file. The GUI seems to be handling the data at some level; preprocessing works well and the visual traces of the analog clearly show spikes.

Here is an image of what the GUI looks like:

This error occurs when trying to 'Spikesort'. I have verified that there are spikes on all channels, and that the data seems to be properly formatted in the .dat file. I have tried variety of thresholds, both negative and positive and have receive the same error.

[nicolomeneghetti]

Dear PutNampt,

I have solved the same issue by simply changing the channel map configuration. Specifically, by using the [new] option of 'Select Probe Layout' of the GUI I have created a custom made channel map of my 16 channels linear probe with 50um of inter electrode spacing as so (see attached screenshot):

[gaito]

Hello, I was just commenting to see if anyone got the aforementioned issue resolved? I am getting it as well!

Error using gpuArray/subsref Subscript indices must either be real positive integers or logicals.

Error in extractTemplatesfromSnippets (line 56) wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));

Error in extract_spikes (line 42) [wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

Error in ksGUI/runSpikesort (line 786) [obj.rez, st3, tF] = extract_spikes(obj.rez);

Error in ksGUI>@(~,~)obj.runSpikesort() (line 337) 'Callback', @(~,~)obj.runSpikesort());

Error while evaluating UIControl Callback.

[bjslater13]

My issue had to do with the path referencing the incorrect function with the same name.

On Fri, Jul 9, 2021, 10:17 AM gaito @.***> wrote:

Hello, I was just commenting to see if anyone got the aforementioned issue resolved? I am getting it as well!

Error using gpuArray/subsref Subscript indices must either be real positive integers or logicals.

Error in extractTemplatesfromSnippets (line 56) wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));

Error in extract_spikes (line 42) [wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

Error in ksGUI/runSpikesort (line 786) [obj.rez, st3, tF] = extract_spikes(obj.rez);

Error in ksGUI>@(,)obj.runSpikesort() (line 337) 'Callback', @(,)obj.runSpikesort());

Error while evaluating UIControl Callback.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MouseLand/Kilosort/issues/335#issuecomment-877222440, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFSJAXYZDPIREV5V34UQLZLTW4AHBANCNFSM4XPZVXIA .

[gaito]

Which function was it - dd? @bjslater13

[bjslater13]

I had a previous installation of kilosort it was referencing. I can't recall the exact function. I ended up resetting the path and adding things one by one.

On Fri, Jul 16, 2021, 11:28 AM gaito @.***> wrote:

Which function was it? @bjslater13 https://github.com/bjslater13

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MouseLand/Kilosort/issues/335#issuecomment-881532839, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFSJAXYE7BIYJNM43CM6C4DTYBF3BANCNFSM4XPZVXIA .

[nikolaskaralis]

In case anybody has this issue again in the future and is reading this thread: There are few different issues discussed above.

For the issue with the error: "Error using randperm K must be less than or equal to N."

The real problem is not having few spikes, so it cannot really be addressed by changing the parameters. The core issue that leads to this error is improper assignment of xcoords and ycoords.

Ensure that your channels have distinct ycoords and it should fix this problem.

[Feikai-Lin]

Does anyone have any update on this?

I also got this problem when I used simulated data generated from main_eMouse_drift.m in the Kilosort3_main folder (with all default configuration/parameters except removing ref channel in make_eMouseChannelMap_3B_short.m). I have checked in GUI that the simulated data was generated and imported properly (with descent amount of spikes etc).

Here are the error messages I got: Error using gpuArray/subsref Subscript indices must either be real positive integers or logicals.

Error extractTemplatesfromSnippets (line 56) wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));

Error datashift2 (line 4) [wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

Error main_eMouse_drift (line 72) rez = datashift2(rez, 1); % last input is for shifting data

While debugging, I find that in extractTemplatesfromSnippets.m: (line 14) fid = fopen(ops.fproc, 'r'); -------> giving that fid = 3 (line 22) dat = fread(fid, [ops.Nchan NT], '*int16'); -------> dat is a large array or zeros.
Since temp_wh.dat is actually quite large (4GB), I am not sure if problems occur in fread or the temp_wh.dat;

I only have ks3 in our workstation (along with Matlab2018b with cuda9.1 on win10). Also I noticed that #320 main_eMouse_drift.m in ks3 hasn't been updated yet. However, the errors I got appeared before learnAndSolve8b.

Any suggestions would be appreciated.

[JRicardo24]

Hi guys, I am having the following problem when running Kilosort 3 on a real dataset of data collected with a 64 channels probe: After compiling the CUDA files with mexGPUall.m, I run the main_kilosort3.m script, it finds the data and performs the preprocessing. However, afterwards, the error shown below appears:

Time 0s. Computing whitening matrix.. Getting channel whitening matrix... Channel-whitening matrix computed. Time 19s. Loading raw data and applying filters... Time 583s. Finished preprocessing 3653 batches. vertical pitch size is 1 horizontal pitch size is 5

    1
    14

0.46 sec, 1 batches, 156 spikes 18.89 sec, 101 batches, 16716 spikes 32.05 sec, 201 batches, 32490 spikes 41.62 sec, 301 batches, 47802 spikes 50.65 sec, 401 batches, 61674 spikes 59.33 sec, 501 batches, 74614 spikes 68.16 sec, 601 batches, 87039 spikes 77.27 sec, 701 batches, 99234 spikes 85.96 sec, 801 batches, 111209 spikes 94.12 sec, 901 batches, 122726 spikes 102.15 sec, 1001 batches, 133689 spikes 110.19 sec, 1101 batches, 144451 spikes 118.38 sec, 1201 batches, 155103 spikes 126.45 sec, 1301 batches, 165466 spikes 134.45 sec, 1401 batches, 176008 spikes 142.53 sec, 1501 batches, 186311 spikes 150.64 sec, 1601 batches, 196413 spikes 158.69 sec, 1701 batches, 206679 spikes 167.08 sec, 1801 batches, 214630 spikes 175.23 sec, 1901 batches, 221979 spikes 183.26 sec, 2001 batches, 228770 spikes 191.29 sec, 2101 batches, 234916 spikes 199.42 sec, 2201 batches, 240615 spikes 207.47 sec, 2301 batches, 245950 spikes 216.33 sec, 2401 batches, 251261 spikes 224.54 sec, 2501 batches, 256356 spikes 232.70 sec, 2601 batches, 261358 spikes 240.86 sec, 2701 batches, 266261 spikes 249.05 sec, 2801 batches, 271129 spikes 257.45 sec, 2901 batches, 276076 spikes 266.29 sec, 3001 batches, 281330 spikes 275.68 sec, 3101 batches, 287202 spikes 285.14 sec, 3201 batches, 292475 spikes 294.48 sec, 3301 batches, 297712 spikes 303.82 sec, 3401 batches, 302829 spikes 311.84 sec, 3501 batches, 310795 spikes 319.64 sec, 3601 batches, 313882 spikes 323.63 sec, 3653 batches, 313983 spikes time 856.61, Shifted up/down 3653 batches. Error using gpuArray/subsref Subscript indices must either be real positive integers or logicals.

Error in extractTemplatesfromSnippets (line 56) wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));

Error in extract_spikes (line 42) [wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

Error in main_kilosort3 (line 40) [rez, st3, tF] = extract_spikes(rez);

Any suggestions? Already tried lowering the ops.spkTh to -4 as @marius10p suggested above, but it hasn't solved it. The compiler in use is Microsoft Visual C++ 2017 in Matlab R2020a version, on a NVIDIA TITAN V GPU and with NVIDIA CUDA 10.1 version.