search

alihaydaroglu / suite3d

Fast, accurate, volumetric cell detection. Developed for Light Beads Microscopy, usable for other volumetric 2P. In development
6 stars 0 forks source link

Crash in sweep_params for correlation map #55

Closed oterocoronel closed 11 months ago

oterocoronel commented 11 months ago

When trying to run the sweep_params like this:

params_to_sweep = {
    'intensity_thresh' : (0.2, 0.3),
    #'n_svd_comp' : (50,), # if you have multiple values here, make sure you pass mov=svd_info
    'conv_filt_xy': (0.7,1.0,),
    'npil_filt_xy': (5.0,),
    'conv_filt_z' : (1.0,),
    'npil_filt_z' : (1.5,),
    'sdnorm_exp' : (0.7,0.8)
}

sweep_summary = job.sweep_params(params_to_sweep, mov = mov,testing_dir_tag='sweep-full',
                           do_vmap=True, all_combinations=True, n_test_iters = 1)

It crashed:

Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full with tag sweep-full
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00000 with tag sweep-full-comb_00000
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00001 with tag sweep-full-comb_00001
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00002 with tag sweep-full-comb_00002
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00003 with tag sweep-full-comb_00003
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00004 with tag sweep-full-comb_00004
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00005 with tag sweep-full-comb_00005
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00006 with tag sweep-full-comb_00006
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00007 with tag sweep-full-comb_00007
   Saving summary for 8 combinations to E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep_summary.npy
Running combination 01/08Total Used: 021.926 GB, Virtual Available: 169.903 GB, Virtual Used: 021.866 GB, Swap Used: 000.060 GB
   Summary dict size: 00 GB
      Combination params: comb00000-params-intensity_thresh_0.200-conv_filt_xy_0.700-npil_filt_xy_5.000-conv_filt_z_1.000-npil_filt_z_1.500-sdnorm_exp_0.700
      Saving to tag sweep-full-comb_00000 at E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00000
   Saved a copy of params at E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00000
   Updated main params file
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00000\iters with tag sweep-full-comb_00000-iters
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00000\mov_sub with tag sweep-full-comb_00000-mov_sub
   Saving mov_sub to E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\sweep-full\sweep-full-comb_00000\mov_sub
   Adjusting temporal hpf to 500 to evenly divide 500 frames
   Using conv_filt: gaussian, 1.00, 0.70
   Using np_filt: unif, 1.50, 5.00
   Created files and dirs for 5 batches
   Running only 1 iters
      Running batch 1 of 1
         Will process 500 frames (0-500, t_batch_size: 500)
      Loaded and swapped, idx 0 to 500
      Calculating corr map
         Rolling mean filter
         Stdev over time
         Sharr creation
         Sub and conv
Exception in thread Thread-112:
Traceback (most recent call last):
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\multiprocessing\connection.py", line 879, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\multiprocessing\connection.py", line 811, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 72
oterocoronel commented 11 months ago

Also happened in calculate_corr_map:

   Updated main params file
   Created dir E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\mov_sub with tag mov_sub
   Saving mov_sub to E:\Data\Curly\20231202d\195518tUTC_Max15_depth300um_fov2044x2000um_res2p00x2p00umpx_fr03p348Hz_pow379p9mW\s3d-Curly_20231202d_380mW\mov_sub
   Adjusting temporal hpf to 500 to evenly divide 500 frames
   Using conv_filt: gaussian, 1.00, 1.00
   Using np_filt: unif, 1.50, 5.00
   Created files and dirs for 5 batches
      Running batch 1 of 5
         Will process 500 frames (0-500, t_batch_size: 500)
      Loaded and swapped, idx 0 to 500
      Calculating corr map
         Rolling mean filter
         Stdev over time
         Sharr creation
         Sub and conv
Exception in thread Thread-5:
Traceback (most recent call last):
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\multiprocessing\connection.py", line 879, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\Freiwald\miniforge3\envs\suite3d-gpu\lib\multiprocessing\connection.py", line 811, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 72
oterocoronel commented 11 months ago

I forgot to change the job.params['n_proc_corr'] to something lower than the number of cores I have. That fixed it.