out of memory issue(zeros function for binned_spikes_depth)

[geonung]

Hi, thanks for your demo code, it is really helpful with SHARP-Track. I recorded mouse neural activity(~30min) using 1 Neuropixels probe with workstation which has 64GB RAM. However, I have trouble to solve 'out of memory' error.

Here is the code and error output:

binned_spikes_depth = zeros(length(unique_depths),length(corr_edges)-1);
for curr_depth = 1:length(unique_depths)
    binned_spikes_depth(curr_depth,:) = histcounts(spike_times( ...
        ismember(spike_templates,find(depth_group == unique_depths(curr_depth)))), ...
        corr_edges);
end

>>> Out of memory.
Related documentation

Size of 'spike_templates' = 62391351(double), and 'corr_edges' = 15617103700(double) Should I size up RAM or allocate datasets? Thanks,

[petersaj]

Hi Youn - this is for the function AP_align_probe_histology? Great if it's useful, just a heads up that that's pretty unrefined and it'll probably be replaced by another tool being developed by the IBL (see demo here https://www.youtube.com/watch?v=JWzj_vZYr-g&feature=youtu.be). Would be happy to hear how it works for you though.

64GB should be good, that 'corr_edges' seems too big: by default it should be time bins in 10ms increments from the first to the last spike, so for 30 min you should be getting ~180k values instead of 15 billion which is what yours is.

Maybe check the step where 'corr_edges' is created in line 28: what are the min and max spike times, are they weird values?

[geonung]

Hi, peters. Sorry for late answer for using IBL demo code and checking my error in AP_align_probe_histology.

Firstly, my data size was 'spike_templates' = 6,239,135 x 1 and 'corr_edges' = 1 x 5,617,103,700(not above value.. sorry.) I checked min and max spike times, and each are 114 and 56,171,151.

Is there a any problem here that i added code for change 'uint32'(spike_templates) and 'uint64'(spike_times) to 'double' data format?

spike_times = double(readNPY('C:\spike_times.npy'));
spike_templates = double(readNPY('C:\spike_templates.npy'))

Thanks for your kind instruction and i'm sorry for my explaining skills, actually i'm a beginner of this area. Happy holiday..!!

[petersaj]

Ah it's probably because your _spiketimes are in integers but it's expecting time in seconds. So you could either convert them into seconds however you normally do it, or for a quick fix just divide by your sampling rate (probably 30k) since this is just getting a correlation of MUA rather than aligning to anything

[geonung]

Thanks, after dividing by sampling_rate(30k), then the problem is solved and works well :) 👍