MouseLand / Kilosort

Fast spike sorting with drift correction
https://kilosort.readthedocs.io/en/latest/
GNU General Public License v3.0
478 stars 248 forks source link

out of memory error in run_pursuit in kilosort3 #442

Closed McKenzieNeuro closed 8 months ago

McKenzieNeuro commented 3 years ago

I am receiving the same error as issue #319 (which has been closed). I just cloned the repo today (Sept 1 2021)

any tips on where to start debugging?

thank you sam


time 0.00, GROUP 1/12, units 0 time 277.47, GROUP 6/12, units 54 Error using * Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'.

Error in bimodal_pursuit (line 12) clp = clp u; % s(ipc, ipc);

McKenzieNeuro commented 3 years ago

I worked on this a bit more and I can begin to see that the issue is that

on line 88 of template_learning.m

kid = run_pursuit(dd, nlow, rmin, n0, wroll, ss(tin), use_CCG);

the variable dd is huge

size(dd) = [ 2473220 6 16]

this looks like a 'dd' is a (large) subselection of data but it is tough to follow how that is done without knowing what is stored in the second column of st3 which is defined in the cuda code. intuitively it seems like it should be about the anatomical localization of the largest spike and I wonder if there is still some channel sorting issue, like in #319