Broadcast error Full_demo

Sihao commented 1 year ago

I am getting this broadcast error. Occurs after registering last batch

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/slu/anaconda3/envs/s2p-lbm/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/slu/anaconda3/envs/s2p-lbm/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/iter_step.py", line 430, in fuse_and_save_reg_file
    mov_fused = fuse_movie(mov, n_skip, centers, shift_xs)
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/iter_step.py", line 411, in fuse_movie
    mov_fused[zidx, :, :, curr_x_new: curr_x_new + wid - n_skip] = \
ValueError: could not broadcast input array from shape (92,584,125) into shape (92,584,124)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/job.py", line 639, in fuse_registered_movie
    fused_files = p.starmap(fuse_and_save_reg_file, [(
  File "/home/slu/anaconda3/envs/s2p-lbm/lib/python3.8/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/home/slu/anaconda3/envs/s2p-lbm/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/slu/anaconda3/envs/s2p-lbm/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/slu/anaconda3/envs/s2p-lbm/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/iter_step.py", line 430, in fuse_and_save_reg_file
    mov_fused = fuse_movie(mov, n_skip, centers, shift_xs)
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/iter_step.py", line 411, in fuse_movie
    mov_fused[zidx, :, :, curr_x_new: curr_x_new + wid - n_skip] = \
ValueError: could not broadcast input array from shape (92,584,125) into shape (92,584,124)

alihaydaroglu commented 1 year ago

I think this happens when the xy shifts for some planes are estimated as some very large numbers because they are outside of the brain. Can you share the "LBM shift between planes" plot?

Quick fix: job.params['fuse_crop'] = ((0,22), (None, None), (None,None)) where replace 22 with the last good plane you have inside the brain for which there is a good estimate for the xy shifts

Sihao commented 1 year ago

LBM_shifts_between_planes

alihaydaroglu commented 1 year ago

This doesn't look very good... Maybe try to use more files/frames during initialization?

oterocoronel commented 1 year ago

Strong crosstalk from deeper to shallower planes could be the reason for this. It looks like the shifts from planes 0-to-14 are being repeated for planes 15-to-29

alihaydaroglu commented 1 year ago

yeah that is a good point - might be the reason. Here's what I'd suggest:

To keep analyzing without re-initializing, try to re-fuse the movie with job.params['fuse_crop'] = ((0,14), (None, None), (None,None)) to only fuse planes 0-14
If you want to analyze planes 15+, verify the crosstalk estimation went well (you can share the crosstalk plot from job.show_summary_plots() here). Based on what that looks like I can suggest how to change the parameters

Sihao commented 1 year ago

Ok will try, thanks for the pointers! This is Jeff's 5x6mm from the paper by the way

alihaydaroglu commented 1 year ago

i think for data acquired before maximum ez (jeff's data), crosstalk estimation wont work as well, because the crosstalk was symmetric (plane 1 leaked into plane 15, plane 15 leaked into plane 1) whereas in maximum ez the leakage only goes in one direction.

jchong-rock commented 1 year ago

Hi, I am having a similar problem at the start of registration when running on the AH007_ROI3p33umNOSCREEN-400um_00001_00004.tif file provided in the Dropbox link. When trying to register plane 0 I get: ValueError: operands could not be broadcast together with shapes (100,1202,1015) (1256,955) (1256,955)

I have tried the quick fix you suggested to Sihao however this does not do anything to solve the issue. I am using the latest commit from the dev branch.

Full traceback:

Loaded 1 files, total 6.82 GB
   Batch 0 IO thread joined
         After IO thread joinTotal Used: 078.104 GB, Virtual Available: 676.498 GB, Virtual Used: 077.826 GB, Swap Used: 000.278 GB
   Subtracting min vals to enfore positivity
         After Sharr creation:Total Used: 085.246 GB, Virtual Available: 669.356 GB, Virtual Used: 084.968 GB, Swap Used: 000.278 GB
      Subtracting plane 1 from 16
      Subtracting plane 2 from 17
      Subtracting plane 3 from 18
      Subtracting plane 4 from 19
      Subtracting plane 5 from 20
      Subtracting plane 6 from 21
      Subtracting plane 7 from 22
      Subtracting plane 8 from 23
      Subtracting plane 9 from 24
      Subtracting plane 10 from 25
      Subtracting plane 11 from 26
      Subtracting plane 12 from 27
      Subtracting plane 13 from 28
      Subtracting plane 14 from 29
   Registering Batch 0
         Before Reg:         Total Used: 085.116 GB, Virtual Available: 669.486 GB, Virtual Used: 084.838 GB, Swap Used: 000.278 GB

      Registering plane 0
Error occured in iteration 0
Traceback (most recent call last):
  File "/lustre/fs4/home/jchong/s2p-lbm/suite3d/iter_step.py", line 742, in register_dataset
    all_offsets = register_mov(mov,refs_and_masks, all_ops, log_cb)
  File "/lustre/fs4/home/jchong/s2p-lbm/suite3d/iter_step.py", line 337, in register_mov
    mov3d[plane_idx], ym, xm, cm, ym1, xm1, cm1 = register.register_frames(
  File "/lustre/fs4/home/jchong/s2p-lbm/suite2p/suite2p/registration/register.py", line 248, in register_frames
    data=rigid.apply_masks(data=fsmooth, maskMul=maskMul, maskOffset=maskOffset),
  File "/lustre/fs4/home/jchong/s2p-lbm/suite2p/suite2p/registration/rigid.py", line 45, in apply_masks
    return addmultiply(data, maskMul, maskOffset)
ValueError: operands could not be broadcast together with shapes (100,1202,1015) (1256,955) (1256,955) 

      Found dir /ru-auth/local/home/jchong/s2p-lbm/demos/Coconut/demo-out/s3d-Coconut-Demo/registered_fused_data with tag registered_fused_data
   Saving to /ru-auth/local/home/jchong/s2p-lbm/demos/Coconut/demo-out/s3d-Coconut-Demo/registered_fused_data
   Saved a copy of params at /ru-auth/local/home/jchong/s2p-lbm/demos/Coconut/demo-out/s3d-Coconut-Demo/registered_fused_data
   Updated main params file
   Cropping: ((0, 20), (None, None), (None, None))
CPU times: user 14.9 s, sys: 26.8 s, total: 41.7 s
Wall time: 1min 18s

alihaydaroglu commented 1 year ago

FUll_Demo runs cleanly now on my end with the sample data on dropbox, please try it out @Sihao @jchong-rock

alihaydaroglu / suite3d

Broadcast error Full_demo #30