Closed timsainb closed 2 months ago
Hi @timsainb
Welcome back! Can you try to profile the call to the save()
function alone? Not sure this is related to Kilosort at all!
recording = recording.save(folder="some-test-folder", progress_bar=True, verbose=True)
I think this is related to:
https://github.com/SpikeInterface/spikeinterface/pull/1602
Or at least it should alleviate the writing data issue.
In the current implementation there is a casting that makes a copy of the files doubling the memory usage in every chunk. Could you try the branch above?
@timsainb How are you measuring ram usage? What is your recording (i.e. spikeglx? openephys?)?
We have a lack of activity for this one of over a year. Did you still want to work on this @h-mayorquin ?
OP never answered what format they were using or if they were applying pre-processing. The information is insufficient to diagnose the issue. Let's close it, if OP wants to provide the information and is still interested on this he can open it again.
Hey All,
I'm recently using spikeinterface again after having not used it for about a year.
In the previous version, you would pass the parameters e.g. "n_jobs_bin": and "total_memory" directly to run_sorters.
In the new version, we do something like:
Then I run
This doesn't seem to work for me though. For example, with the above parameters, I get the following:
Figure: (x-axis is seconds, y axis is RAM usage in MB). I ran two sorts, one with a 1.5hr dataset, the other 3.5 hours. I had an allocation of 20GB ram, and the second sort crashed. The initial jumps in ram usage are spikeinterface writing the data file. Then you see kilosort 2.5 running. Both the writer and the kilosort job use well above the allotted RAM.
I tried bumping the NT batch size down to 8102464 (1/8 the size) with the same larger dataset and I get only a slight reduction in RAM usage:
So my question is, how can I control the RAM utilization better?