search

cortex-lab / KiloSort

GPU code for spike sorting
GNU General Public License v2.0
175 stars 100 forks source link

Out of Memory during re-ordering by depth #197

Open brendonw1 opened 5 years ago

brendonw1 commented 5 years ago

We are recording from linear single shank probes with 64 channels each 20um spaced from each other over about 6-8 hours and are getting out of memory errors. I wondered if anything can be done to allow us to process large/long files that cannot be broken into smaller groups of channels.

We get the error "Error running kilosort! Call to Real Single Symmetric EIG on the GPU failed with CUDA error status out of memory".

This about 30min after the line "Pre-clustering to re-order batches by depth" is displayed in the GUI message log after about 18500 batches.

Can anything be done?

Thank you

brendonw1 commented 5 years ago

Wanted to bump this in case there is a way to address it... is there likely to be a way to do so?

Yliew6-gt commented 5 years ago

I also ran into the exact same error message. Tried to ran using the GUI as well as master script but both are not working. I have also tried reducing the batch size number (ops.NT on config file) but still ran into the same error message. What other parameter should I change? Have you figured it out? @brendonw1

brendonw1 commented 5 years ago

We haven't found a solution unfortunately. I wonder if @nsteinme or @marius10p might have ideas?

ckemere commented 4 years ago

Did you ever find a solution to this? Is it possible to disable GPU and only use CPU?

timsainb commented 3 years ago

I ran into this error as well, using SpikeInterface with Kilosort2 on a 32 channel ~24hr recording with a Titan XP. No issues with the 12hr recording. I'll probably break the recording up into channels/smaller time intervals, but here's my error, for visibility. 

time 883.57, compared 26501 / 26760 batches 

--------------------------------------------------------------------------------Error using gpuArray/eig

Call to Real Single Symmetric EIG on the GPU failed with CUDA error status: out of memory.Error using gpuArray/eig

Call to Real Single Symmetric EIG on the GPU failed with CUDA error status: out of memory.
 SpikeSortingError: Spike sorting failed: kilosort2 returned a non-zero exit code. You can inspect the runtime trace in the kilosort2.log of the output folder.'
marius10p commented 3 years ago

This is expected, and no longer a problem with Kilosort 2.5. This is the wrong repo though.