sweep_parameters uses large amounts of RAM for large FOV recordings

[oterocoronel]

When running job.sweep_params(params_to_sweep, svd_info, ... ) for a 1000x1000 pix recording and 300 SVD components, RAM usage goes to ~400 GB and it takes a very long time to run.

[2023-07-21 08:48:02][01]    Will reconstruct SVD movie on-the-fly from  with 300 components
[2023-07-21 09:31:59][01]    Loaded spatial components in 2636.69 seconds, 195.26 GB
[2023-07-21 09:32:01][02]       Running batch 1 of 1
[2023-07-21 09:32:01][02]       Reconstructing from svd
[2023-07-21 09:32:21][03]          Loaded Us
[2023-07-21 09:32:21][04]             Reconstructing block 0
...
[2023-07-21 09:49:37][04]             Reconstructing block 1075
[2023-07-21 09:51:35][02]       Reconstructed in 1173.77 seconds
[2023-07-21 09:51:35][02]       Calculating corr map
[2023-07-21 09:51:35][03]          Rolling mean filter
[2023-07-21 09:52:09][03]          Stdev over time
[2023-07-21 09:55:36][03]          Sharr creation
[2023-07-21 09:56:17][03]          Sub and conv
[2023-07-21 10:03:49][03]          Vmap
[2023-07-21 10:04:41][02]       Calculated corr map in 786.36 seconds

When running job.sweep_params(params_to_sweep, mov = svd_info, ...), the issue persists. Haven't finished yet but it is using +340 GB RAM and it has been stuck at this step for 20+ minutes (and I assume there will be more batches to process? it is trying to use the whole movie?):

[2023-07-21 14:15:58][01]    Will reconstruct SVD movie on-the-fly from  with 300 components
[2023-07-21 14:15:58][01]    Created files and dirs for 12 batches
[2023-07-21 14:15:58][01]    Running only 1 iters
[2023-07-21 14:15:58][02]       Running batch 1 of 12
[2023-07-21 14:15:58][03]          Reconstructing from svd

Parameters:

job.params['t_batch_size'] = 300 # 

job.params['n_proc_corr'] = 19
job.params['n_svd_comp'] = 200
job.params['intensity_thresh'] = 0.25

params_to_sweep = {
    'intensity_thresh' : (0.25, 0.35),
    'n_svd_comp' : (300,200,400),
}

n_test_iters = 1

[alihaydaroglu]

When you do job.calculate_corrmap outside of the sweep, does that also have the same issue?

[oterocoronel]

Trying that now. With a batch of 100 frames:

   Will reconstruct SVD movie on-the-fly from  with 300 components
   Created files and dirs for 36 batches
      Running batch 1 of 36
         Reconstructing from svd
(1092, 100, 160000)
         Reconstructed in 607.30 seconds
      Loaded and swapped, idx 0 to 100
      Calculating corr map
         Rolling mean filter
         Stdev over time
         Sharr creation
         Sub and conv
         Vmap
   Calculated corr map in 156.99 seconds

RAM usage did not go as high (~100 GB)

[alihaydaroglu]

OK, if you try the sweep with job.sweep_params(params_to_sweep, mov = svd_info, ...) and a small batchsize of 100, it should run the exact same code as job.calculate_corrmap above, so it won't use crazy amounts of memory. You can change n_test_iters to 4 for the sweep to use 400 frames. I think that should work faster in your case?

I will change the parameter interface to be less confusing in the future, but for now:

• if you want less memory usage, but are OK to pay the price of some extra I/O time at each iteration, set mov=svd_info as an argument to sweep_params or calculate_corr_map, and set smaller batchsizes
• if you have a lot of RAM (or a smaller FOV size in pixels), set svd_info=svd_info to load and keep all of the spatial components in memory once

As of https://github.com/alihaydaroglu/s2p-lbm/commit/22f6fdf7177e5b9129560305cbf67c1f63557f6d the former is the default in the demo notebook, and I've added a warning if you are using the RAM-greedy option

[oterocoronel]

It seems like n_test_iters is not being used when using mov=svd_info and sweeping, it is trying to go through the whole recording (n_batches * t_batch_size = total_frames). Might be nice to just do one batch for sweeping

[alihaydaroglu]

closing this for now, since small batchsizes and using the less RAM-greedy mode seem to fix the issue.

n_test_iters works as intended, but there are a few misleading print statements that i'll fix to clarify that it's only doing a few batches