Open RichieHakim opened 2 years ago
saramoberg
commented
2 years ago Hi, not a developer but another user. I'm also having similar issues but not due to denoising. I think it's because it's detecting too many ROIs (10-20k ROIs) which makes the dataset very large..... Following this issue for any solutions!
generalciao
commented
2 years ago Which suite2p versions are you (Richard, Sara) using? I see a new v0.11.0 release on pypi from a few days ago, which incorporates changes to allow non-default cellpose 2.0 models to be used, but I think also some refactoring and other changes. Did you update via pip, or are you using the latest github version??
On Thu, Jun 30, 2022 at 4:41 PM Sara Moberg @.***> wrote:
Hi, not a developer but another user. I'm also having similar issues but not due to denoising. I think it's because it's detecting too many ROIs (10-20k ROIs) which makes the dataset very large..... Following this issue for any solutions!
— Reply to this email directly, view it on GitHub https://github.com/MouseLand/suite2p/issues/832#issuecomment-1171306443, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGJQVJQ325CYKKUNXQJQBP3VRWWX7ANCNFSM5WN2EHHA . You are receiving this because you are subscribed to this thread.Message ID: @.***>
saramoberg
commented
2 years ago Thanks a lot! I just updated today via pip and now I have v0.11.1
But I also tried using the github version because I saw someone had some similar issues with Cellpose. Now, however, I'm getting some TypeError with the first ROI mask. I created a new issue for this problem though...
just-meng
commented
2 years ago Same problem here. I have installed the following versions:
cellpose 2.0.5 pypi_0 pypi
python 3.9.12 h6244533_0
suite2p 0.11.0 pypi_0 pypiI have also tried reinstalling using python=3.8, and pip install git+https://github.com/MouseLand/suite2p.git. Still get the same error. Anything else I could try?
generalciao
commented
2 years ago Richard, at first glance it looks like denoise.py is trying to work with 1320 blocks, which seems like way too many to me (even for your large images). There's an earlier indication in your log that something may be amiss with the binning: Binning movie in chunks of length 07 Binned movie [2627,4660,512] in 242.83 sec.
For me (frame size 796 wide, 512 tall) and ~30k frames, that same statement (with an earlier suite2p version) reads: Binning movie in chunks of length 19 Binned movie [1560,494,774] in 154.21 sec.
The 2nd and 3rd values refer to the Lx and Ly dimensions of the cropped frame (796 pixels -> 774, 512 pixels -> 494, in my case), and the 1st value refers to the # of frames. In my case the # frames (1560) times chunk length (19) roughly equals the total # of frames (30k).
In your case, the # frames (2627) times chunk length (07) also roughly equals the total # of frames you report (18k). However, you state that your frames are each 1536x512 pixels. So the second dimension of the binned movie, 4660, seems to be >3x too large. This could also explain how you end up with such a large number of blocks to be denoised.
Maybe not important, but the 512 dimension (typically # of lines in the slow scanner direction) is in the 3rd position for your log, and in the 2nd position for my log output.
If you look at the registered frames that are saved to disk (as .bin or .tif), what size are they? Close to 1536x512 or are they much wider (4500-5000 pixels)?
On Fri, May 20, 2022 at 2:18 AM Richard Hakim @.***> wrote:
After upgrading suite2p, I am seeing very high memory requirements for the PCA denoising step. I'm not sure how to remedy this. Input movies are 1536x512, and there are roughly 18,000 frames.
[image: image] https://user-images.githubusercontent.com/9909734/169424182-71e98a16-bac5-4072-998c-a57955055cc6.png {'data_path': ['E:/data/Ally/AEG22/2022_05_13'], 'subfolders': [], 'save_path0': 'E:/data/Ally/AEG22/2022_05_13', 'fast_disk': 'F:/', 'input_format': 'tif'} tif Found 19 tifs - converting to binary 400 frames of binary, time 4.50 sec. 800 frames of binary, time 8.97 sec. 1200 frames of binary, time 20.05 sec. 1600 frames of binary, time 28.92 sec. 2000 frames of binary, time 37.81 sec. 2400 frames of binary, time 54.29 sec. 2800 frames of binary, time 64.92 sec. 3200 frames of binary, time 82.04 sec. 3600 frames of binary, time 92.32 sec. 4000 frames of binary, time 104.05 sec. 4400 frames of binary, time 122.35 sec. 4800 frames of binary, time 134.74 sec. 5200 frames of binary, time 153.38 sec. 5600 frames of binary, time 164.60 sec. 6000 frames of binary, time 175.40 sec. 6400 frames of binary, time 192.75 sec. 6800 frames of binary, time 203.53 sec. 7200 frames of binary, time 221.04 sec. 7600 frames of binary, time 232.60 sec. 8000 frames of binary, time 244.43 sec. 8400 frames of binary, time 262.59 sec. 8800 frames of binary, time 274.37 sec. 9200 frames of binary, time 292.54 sec. 9600 frames of binary, time 304.08 sec. 10000 frames of binary, time 315.93 sec. 10400 frames of binary, time 333.17 sec. 10800 frames of binary, time 343.48 sec. 11200 frames of binary, time 357.21 sec. 11600 frames of binary, time 365.87 sec. 12000 frames of binary, time 374.53 sec. 12400 frames of binary, time 389.04 sec. 12800 frames of binary, time 399.33 sec. 13200 frames of binary, time 416.15 sec. 13600 frames of binary, time 426.41 sec. 14000 frames of binary, time 436.60 sec. 14400 frames of binary, time 453.46 sec. 14800 frames of binary, time 463.54 sec. 15200 frames of binary, time 479.24 sec. 15600 frames of binary, time 489.59 sec. 16000 frames of binary, time 499.54 sec. 16400 frames of binary, time 515.07 sec. 16800 frames of binary, time 524.28 sec. 17200 frames of binary, time 537.61 sec. 17600 frames of binary, time 546.55 sec. 18000 frames of binary, time 555.22 sec. 18400 frames of binary, time 566.85 sec. time 569.19 sec. Wrote 18504 frames per binary for 1 planes
PLANE 0 <<<<<<<<<<<<<<<<<<<<<< NOTE: not registered / registration forced with ops['do_registration']>1 (no previous offsets to delete) ----------- REGISTRATION registering 18504 frames Reference frame, 560.47 sec. Registered 400/18504 in 211.64s Registered 800/18504 in 423.71s Registered 1200/18504 in 635.89s Registered 1600/18504 in 848.29s Registered 2000/18504 in 1060.19s Registered 2400/18504 in 1272.43s Registered 2800/18504 in 1484.21s Registered 3200/18504 in 1696.51s Registered 3600/18504 in 1908.51s Registered 4000/18504 in 2120.92s Registered 4400/18504 in 2333.33s Registered 4800/18504 in 2546.32s Registered 5200/18504 in 2758.64s Registered 5600/18504 in 2971.27s Registered 6000/18504 in 3184.37s Registered 6400/18504 in 3397.04s Registered 6800/18504 in 3610.24s Registered 7200/18504 in 3823.14s Registered 7600/18504 in 4035.51s Registered 8000/18504 in 4248.49s Registered 8400/18504 in 4461.76s Registered 8800/18504 in 4674.79s Registered 9200/18504 in 4887.86s Registered 9600/18504 in 5101.27s Registered 10000/18504 in 5314.00s Registered 10400/18504 in 5526.88s Registered 10800/18504 in 5740.67s Registered 11200/18504 in 5953.53s Registered 11600/18504 in 6166.45s Registered 12000/18504 in 6379.40s Registered 12400/18504 in 6592.21s Registered 12800/18504 in 6805.41s Registered 13200/18504 in 7018.34s Registered 13600/18504 in 7231.25s Registered 14000/18504 in 7444.29s Registered 14400/18504 in 7657.37s Registered 14800/18504 in 7870.39s Registered 15200/18504 in 8083.34s Registered 15600/18504 in 8296.44s Registered 16000/18504 in 8509.47s Registered 16400/18504 in 8722.91s Registered 16800/18504 in 8935.94s Registered 17200/18504 in 9148.82s Registered 17600/18504 in 9362.38s Registered 18000/18504 in 9575.92s Registered 18400/18504 in 9788.93s added enhanced mean image ----------- Total 10462.72 sec ----------- REGISTRATION STEP 2 (making mean image (excluding bad frames) registering 18504 frames NOTE: user reference frame given Registered 400/18504 in 210.96s Registered 800/18504 in 422.37s Registered 1200/18504 in 633.43s Registered 1600/18504 in 845.64s Registered 2000/18504 in 1057.51s Registered 2400/18504 in 1268.82s Registered 2800/18504 in 1479.54s Registered 3200/18504 in 1687.79s Registered 3600/18504 in 1897.20s Registered 4000/18504 in 2108.36s Registered 4400/18504 in 2320.59s Registered 4800/18504 in 2532.27s Registered 5200/18504 in 2744.45s Registered 5600/18504 in 2956.57s Registered 6000/18504 in 3169.03s Registered 6400/18504 in 3380.89s Registered 6800/18504 in 3593.12s Registered 7200/18504 in 3805.62s Registered 7600/18504 in 4017.94s Registered 8000/18504 in 4230.28s Registered 8400/18504 in 4442.56s Registered 8800/18504 in 4654.64s Registered 9200/18504 in 4867.07s Registered 9600/18504 in 5079.59s Registered 10000/18504 in 5291.96s Registered 10400/18504 in 5503.90s Registered 10800/18504 in 5716.28s Registered 11200/18504 in 5928.95s Registered 11600/18504 in 6141.51s Registered 12000/18504 in 6353.94s Registered 12400/18504 in 6566.48s Registered 12800/18504 in 6778.91s Registered 13200/18504 in 6992.02s Registered 13600/18504 in 7204.36s Registered 14000/18504 in 7417.04s Registered 14400/18504 in 7629.28s Registered 14800/18504 in 7841.62s Registered 15200/18504 in 8054.01s Registered 15600/18504 in 8266.90s Registered 16000/18504 in 8479.56s Registered 16400/18504 in 8692.06s Registered 16800/18504 in 8904.73s Registered 17200/18504 in 9117.15s Registered 17600/18504 in 9329.96s Registered 18000/18504 in 9542.59s Registered 18400/18504 in 9755.04s added enhanced mean image ----------- Total 20337.11 sec Registration metrics, 285.51 sec. NOTE: applying default C:\Users\scanimage.suite2p\classifiers\classifier_user.npy ----------- ROI DETECTION Binning movie in chunks of length 07 Binned movie [2627,4660,512] in 242.83 sec. ERROR<<< Traceback (most recent call last): File "C:\Users\scanimage\miniconda3\envs\suite2p\lib\runpy.py", line 193, in
run_module_as_main ERROR<<< "main", mod_spec) File "C:\Users\scanimage\miniconda3\envs\suite2p\lib\runpy.py", line 85, in run_code ERROR<<< exec(code, run_globals) File "C:\Users\scanimage\miniconda3\envs\suite2p\lib\site-packages\suite2p_main.py", line 78, in ERROR<<< main() File "C:\Users\scanimage\miniconda3\envs\suite2p\lib\site-packages\suite2p_main.py", line 71, in main ERROR<<< run_s2p(ops, db) File "C:\Users\scanimage\miniconda3\envs\suite2p\lib\site-packages\suite2p\run_s2p.py", line 428, in run_s2p ERROR<<< op = run_plane(op, ops_path=ops_path) File "C:\Users\scanimage\miniconda3\envs\suite2p\lib\site-packages\suite2p\run_s2p.py", line 239, in run_plane ERROR<<< ops, stat = detection.detect(ops=ops, classfile=classfile) File "C:\Users\scanimage\miniconda3\envs\suite2p\lib\site-packages\suite2p\detection\detect.py", line 41, in detect ERROR<<< n_comps_frac = 0.5) File "C:\Users\scanimage\miniconda3\envs\suite2p\lib\site-packages\suite2p\detection\denoise.py", line 22, in pca_denoise ERROR<<< block_re = np.zeros((nblocks, nframes, Lyb*Lxb)) ERROR<<< MemoryError>>>ERROR<<< : Unable to allocate 106. GiB for an array with shape (1320, 2627, 4096) and data type float64 Interrupted by error (not finished)
— Reply to this email directly, view it on GitHub https://github.com/MouseLand/suite2p/issues/832, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGJQVJQHZKFF6HRIETAQ3SLVK3K37ANCNFSM5WN2EHHA . You are receiving this because you are subscribed to this thread.Message ID: @.***>
just-meng
commented
2 years ago Hi generalciao,
In my case the dimensions seem to be correct. I've got 14700 frames in total, of the size 512x512. The registered tif has the size 500x512x512.
Binning movie in chunks of length 02 Binned movie of size [7350,510,508] created in 61.15 sec.
This seems to be alright. Any other idea? See the complete error message below.
Thanks, Meng
{'data_path': ['E:/Data/Two-photon/Preliminary_Psychedelics/220303A_Ai9xSim/22-04-24_DMT/DMT_1'], 'subfolders': [], 'save_path0': 'E:/Data/Two-photon/Preliminary_Psychedelics/220303A_Ai9xSim/22-04-24_DMT/DMT_1', 'fast_disk': 'E:/Data/Two-photon/Preliminary_Psychedelics/220303A_Ai9xSim/22-04-24_DMT/DMT_1', 'input_format': 'tif'} tif Found 49 tifs - converting to binary 6000 frames of binary, time 69.08 sec. 12000 frames of binary, time 140.22 sec. time 174.33 sec. Wrote 14700 frames per binary for 1 planes
PLANE 0 <<<<<<<<<<<<<<<<<<<<<< NOTE: not registered / registration forced with ops['do_registration']>1 (no previous offsets to delete) NOTE: applying default C:\Users\jiame.suite2p\classifiers\classifier_user.npy ----------- REGISTRATION Reference frame, 12.95 sec. Registered 500/14700 in 28.80s Registered 1000/14700 in 49.47s Registered 1500/14700 in 70.92s Registered 2000/14700 in 92.53s Registered 2500/14700 in 113.18s Registered 3000/14700 in 133.78s Registered 3500/14700 in 155.32s Registered 4000/14700 in 177.13s Registered 4500/14700 in 199.15s Registered 5000/14700 in 219.64s Registered 5500/14700 in 240.54s Registered 6000/14700 in 261.67s Registered 6500/14700 in 282.45s Registered 7000/14700 in 302.45s Registered 7500/14700 in 323.29s Registered 8000/14700 in 343.59s Registered 8500/14700 in 367.99s Registered 9000/14700 in 391.93s Registered 9500/14700 in 416.92s Registered 10000/14700 in 439.48s Registered 10500/14700 in 460.81s Registered 11000/14700 in 481.29s Registered 11500/14700 in 501.19s Registered 12000/14700 in 521.86s Registered 12500/14700 in 542.06s Registered 13000/14700 in 563.30s Registered 13500/14700 in 583.34s Registered 14000/14700 in 604.82s Registered 14500/14700 in 625.68s Registered 14700/14700 in 633.98s ----------- Total 650.96 sec Registration metrics, 136.46 sec. ----------- ROI DETECTION Binning movie in chunks of length 02 Binned movie of size [7350,510,508] created in 61.15 sec. ERROR<<< Traceback (most recent call last): File "C:\Users\jiame\Anaconda3\envs\suite2p\lib\runpy.py", line 197, in _run_module_as_main ERROR<<< return _run_code(code, main_globals, None, File "C:\Users\jiame\Anaconda3\envs\suite2p\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\jiame\Anaconda3\envs\suite2p\lib\site-packages\suite2p__main.py", line 78, in
main() File "C:\Users\jiame\Anaconda3\envs\suite2p\lib\site-packages\suite2p\ main__.py", line 71, in main run_s2p(ops, db) File "C:\Users\jiame\Anaconda3\envs\suite2p\lib\site-packages\suite2p\run_s2p.py", line 429, in run_s2p op = run_plane(op, ops_path=ops_path) File "C:\Users\jiame\Anaconda3\envs\suite2p\lib\site-packages\suite2p\run_s2p.py", line 292, in run_plane ops = pipeline(f_reg, f_raw, f_reg_chan2, f_raw_chan2, run_registration, ops, stat=stat) File "C:\Users\jiame\Anaconda3\envs\suite2p\lib\site-packages\suite2p\run_s2p.py", line 129, in pipeline ops, stat = detection.detection_wrapper(f_reg, File "C:\Users\jiame\Anaconda3\envs\suite2p\lib\site-packages\suite2p\detection\detect.py", line 92, in detection_wrapper mov = pca_denoise(mov, block_size=[ops['block_size'][0]//2, ops['block_size'][1]//2], File "C:\Users\jiame\Anaconda3\envs\suite2p\lib\site-packages\suite2p\detection\denoise.py", line 22, in pca_denoise block_re = np.zeros((nblocks, nframes, Lyb*Lxb)) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 32.3 GiB for an array with shape (144, 7350, 4096) and data type float64 Interrupted by error (not finished)
saraysoldado
commented
2 years ago Hi everyone,
I was having the same error and could solve it increasing the 'pagefile' of the hardisk where I have my data:
https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type
(See second reply in that link for how to do this in Windows)
I am not sure if this is the best way to go, but may be a temporary fix. Let me know if you have any feedback on this.
Thanks Saray
After upgrading suite2p, I am seeing very high memory requirements for the PCA denoising step. I'm not sure how to remedy this. Input movies are 1536x512, and there are roughly 18,000 frames.