JinghaoLu / MIN1PIPE

A MINiscope 1-photon-based Calcium Imaging Signal Extraction PIPEline.
GNU General Public License v3.0
56 stars 25 forks source link

Memory errors; Batch size parameters? #25

Open gviejo opened 4 years ago

gviejo commented 4 years ago

Hello,

Thanks for the effort in writing that package. I am testing it with 20 GB of data recorded with the UCLA miniscope. I keep stumbling onto memory errors that do not appear if I give only a few GB of data :

Out of memory. Type HELP MEMORY for your options. Error in get_noise_fft (line 32) xdft = fft(tmp, [], 2); Error in preprocess_data (line 48) [sn, psx] = get_noise_fft(m, options); Error in par_init (line 8) P = preprocess_data(m);Error in min1pipe (line 140) [P, options] = par_init(m);`

or

Error using inter_section (line 84) Out of memory attempting to serialize data for transmission. Error in frame_reg (line 107) m = inter_section(m, sttn, se, pixs, scl, sigma_x, sigma_f, sigma_d, maskc); Error in min1pipe (line 108) [m, corr_score, raw_score, scl] = frame_reg(m, imaxy, se, Fsi_new, pixs, scl, sigma_x, sigma_f, sigma_d);

I tested it on a linux machine with matlab R2019b, 32GB memory and a window machine with matlab R2018b, 16 GB memory. Both have more than 100 GB free space on the disk. The linux machine has 31 GB of SWAP memory.

So, is there a way to adjust a batch size parameter somewhere instead of playing with the SWAP memory?

Apologies if the question has been asked before or if it's just trivial, I am not very familiar with matlab.

Thanks for your help

Guillaume

JinghaoLu commented 4 years ago

It seems the original batch calculation results in the batch size that is on the boundary of the memory in your machine, because sometimes you can and sometimes you cannot pass the movement correction session. A simple but temporary solution is that you can go to "batch_compute.m" and replace in line 3 or 7 (based on which system you are using) with a higher denominator and see if it get the stuck points passed. I will need to see if there is a systematic fix for this, or this is something specific in your machine.

gviejo commented 4 years ago

So I went from

memo = unix_mem / 2;

to

memo = unix_mem / 4;

But now, it crashes with this error :

Begin inter-section ... Done data prep Done loop #1/6 Done loop #2/6 Done loop #3/6 Done loop #4/6 Done loop #5/6 Done loop #6/6 Unable to perform assignment because the left and right sides have a different number of elements.

Error in inter_section (line 163) idbatch(i) = idtmp;

Error in frame_reg (line 107) m = inter_section(m, sttn, se, pixs, scl, sigma_x, sigma_f, sigma_d, maskc);

Error in min1pipe (line 109) [m, corr_score, raw_score, scl] = frame_reg(m, imaxy, se, Fsi_new, pixs, scl, sigma_x, sigma_f, sigma_d);

IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down.

Thanks for the help

JinghaoLu commented 4 years ago

Do you mind setting a breakpoint on line 161 of inter_section.m? Then check some related variables, such as idbatch, dfc etc. to see what the problem really is.

Les0me commented 4 years ago

Hi, I have a similar problem ('out of memory on device'), also with 20 GB of data recorded with the UCLA miniscope system. My computer specs are: 256 GB RAM and two Intel Xeon E5-2698v3 each with 16 cores, running with Ubuntu 18.04 and Matlab 2018b.

Many thanks in advance!

demo_min1pipe Overwrite raw .mat file (data)? (y/n)y Begin collecting datasets info Done collecting datasets info, time: 365.1997 Begin data cat 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700 7800 7900 8000 8100 8200 8300 8400 8500 8600 8700 8800 8900 9000 9100 9200 9300 9400 9500 9600 9700 9800 9900 10000 10100 10200 10300 10400 10500 10600 10700 10800 10900 11000 11100 11200 11300 11400 11500 11600 11700 11800 11900 12000 12100 12200 12300 12400 12500 12600 12700 12800 12900 13000 13100 13200 13300 13400 13500 13600 13700 13800 13900 14000 14100 14200 14300 14400 14500 14600 14700 14800 14900 15000 15100 15200 15300 15400 15500 15600 15700 15800 15900 16000 16100 16200 16300 16400 16500 16600 16700 16800 16900 17000 17100 17200 17300 17400 17500 17600 17700 17800 17900 18000 18100 18200 18300 18400 18500 18600 18700 18800 18900 19000 19100 19200 19300 19400 19500 19600 19700 19800 19900 20000 20100 20200 20300 20400 20500 20600 20700 20800 20900 21000 21100 21200 21300 21400 21500 21600 21700 21800 21900 22000 22100 22200 22300 22400 22500 22600 22700 22800 22900 23000 23100 23200 23300 23400 23500 23600 23700 23800 23900 24000 24100 24200 24300 24400 24500 24600 24700 24800 24900 25000 25100 25200 25300 25400 25500 25600 25700 25800 25900 26000 26100 26200 26300 26400 26500 26600 26700 26800 26900 27000 27100 27200 27300 27400 27500 27600 27700 27800 27900 28000 28100 28200 28300 28400 28500 28600 28700 28800 28900 29000 29100 29200 29300 29400 29500 29600 29700 29800 29900 30000 30100 30200 30300 30400 30500 30600 30700 30800 30900 31000 31100 31200 31300 31400 31500 31600 31700 31800 31900 32000 32100 32200 32300 32400 32500 32600 32700 32800 32900 33000 33100 33200 33300 33400 33500 33600 33700 33800 33900 34000 34100 34200 34300 34400 34500 34600 34700 34800 34900 35000 35100 35200 35300 35400 35500 35600 35700 35800 35900 36000 36100 36200 36300 36400 36500 36600 36700 36800 36900 37000 37100 37200 37300 37400 37500 37600 37700 37800 37900 38000 38100 38200 38300 38400 38500 38600 38700 38800 38900 39000 39100 39200 39300 39400 39500 39600 39700 39800 39900 40000 40100 40200 40300 40400 40500 40600 40700 40800 40900 41000 41100 41200 41300 41400 41500 41600 41700 41800 41900 42000 42100 42200 42300 42400 42500 42600 42700 42800 42900 43000 43100 43200 43300 43400 43500 43600 43700 43800 43900 44000 44100 44200 44300 44400 44500 44600 44700 44800 44900 45000 45100 45200 45300 45400 45500 45600 45700 45800 45900 46000 46100 46200 46300 46400 46500 46600 46700 46800 46900 47000 47100 47200 47300 47400 47500 47600 47700 47800 47900 48000 48100 48200 48300 48400 48500 48600 48700 48800 48900 49000 49100 49200 49300 49400 49500 49600 49700 49800 49900 50000 50100 50200 50300 50400 50500 50600 50700 50800 50900 51000 51100 51200 51300 51400 51500 51600 51700 51800 51900 52000 52100 52200 52300 52400 52500 52600 52700 52800 52900 53000 53100 53200 53300 53400 53500 53600 53700 53800 53900 54000 54100 54200 54300 54400 54500 54600 54700 54800 54900 55000 55100 55200 55300 55400 55500 55600 55700 55800 55900 56000 56100 56200 56300 56400 56500 56600 56700 56800 56900 57000 57100 57200 57300 57400 57500 57600 57700 57800 57900 58000 58100 58200 58300 58400 58500 58600 58700 58800 58900 59000 59100 59200 Done data cat, time: 73289.4388 Overwrite neural enhanced & registrated .mat file (data)? (y/n)y Begin dirts clean #1/22 batch Starting parallel pool (parpool) using the 'local' profile ... connected to 32 workers. Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #2500/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #2300/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2600/2695 Done dirts clean #1/22 batch Begin anisotropic diffusion #1/22 batch done frame #700/2695 done frame #900/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #1500/2695 done frame #1200/2695 done frame #500/2695 done frame #800/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #2000/2695 done frame #2200/2695 done frame #1800/2695 done frame #1900/2695 done frame #1700/2695 done frame #2100/2695 done frame #2500/2695 done frame #2300/2695 done frame #2600/2695 done frame #2400/2695 Done anisotropic diffusion #1/22 batch Begin background removal #1/22 batch Done frame #700/2695 Done frame #800/2695 Done frame #900/2695 Done frame #1800/2695 Done frame #1000/2695 Done frame #1900/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #500/2695 Done frame #600/2695 Done frame #1700/2695 Done frame #2200/2695 Done frame #2100/2695 Done frame #2300/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2000/2695 Done background removal #1/22 batch Begin saving Ydebg #1/22 batch Done saving Ydebg #1/22 batch Begin dirts clean #2/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2300/2695 Done frame #2500/2695 Done frame #2400/2695 Done frame #2600/2695 Done dirts clean #2/22 batch Begin anisotropic diffusion #2/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #500/2695 done frame #1500/2695 done frame #800/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #1800/2695 done frame #2000/2695 done frame #2200/2695 done frame #1900/2695 done frame #1700/2695 done frame #2100/2695 done frame #2500/2695 done frame #2300/2695 done frame #2400/2695 done frame #2600/2695 Done anisotropic diffusion #2/22 batch Begin background removal #2/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #1400/2695 Done frame #2200/2695 Done frame #2400/2695 Done frame #2100/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2600/2695 Done background removal #2/22 batch Begin saving Ydebg #2/22 batch Done saving Ydebg #2/22 batch Begin dirts clean #3/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2600/2695 Done dirts clean #3/22 batch Begin anisotropic diffusion #3/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #500/2695 done frame #1500/2695 done frame #800/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #1800/2695 done frame #2000/2695 done frame #2200/2695 done frame #1900/2695 done frame #1700/2695 done frame #2100/2695 done frame #2500/2695 done frame #2300/2695 done frame #2400/2695 done frame #2600/2695 Done anisotropic diffusion #3/22 batch Begin background removal #3/22 batch Done frame #700/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #1400/2695 Done frame #1900/2695 Done frame #2100/2695 Done frame #2200/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2400/2695 Done frame #2600/2695 Done background removal #3/22 batch Begin saving Ydebg #3/22 batch Done saving Ydebg #3/22 batch Begin dirts clean #4/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2300/2695 Done dirts clean #4/22 batch Begin anisotropic diffusion #4/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #1500/2695 done frame #500/2695 done frame #800/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #1800/2695 done frame #2000/2695 done frame #2200/2695 done frame #1700/2695 done frame #1900/2695 done frame #2100/2695 done frame #2300/2695 done frame #2500/2695 done frame #2600/2695 done frame #2400/2695 Done anisotropic diffusion #4/22 batch Begin background removal #4/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #1400/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2300/2695 Done background removal #4/22 batch Begin saving Ydebg #4/22 batch Done saving Ydebg #4/22 batch Begin dirts clean #5/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2200/2695 Done frame #2300/2695 Done frame #2500/2695 Done frame #2400/2695 Done frame #2600/2695 Done dirts clean #5/22 batch Begin anisotropic diffusion #5/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1500/2695 done frame #1200/2695 done frame #500/2695 done frame #800/2695 done frame #1100/2695 done frame #1600/2695 done frame #1400/2695 done frame #1800/2695 done frame #2000/2695 done frame #2200/2695 done frame #1900/2695 done frame #1700/2695 done frame #2100/2695 done frame #2300/2695 done frame #2400/2695 done frame #2500/2695 done frame #2600/2695 Done anisotropic diffusion #5/22 batch Begin background removal #5/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2600/2695 Done background removal #5/22 batch Begin saving Ydebg #5/22 batch Done saving Ydebg #5/22 batch Begin dirts clean #6/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2300/2695 Done frame #2500/2695 Done frame #2400/2695 Done frame #2600/2695 Done dirts clean #6/22 batch Begin anisotropic diffusion #6/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #1500/2695 done frame #500/2695 done frame #800/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #1800/2695 done frame #2000/2695 done frame #2200/2695 done frame #1900/2695 done frame #2100/2695 done frame #1700/2695 done frame #2300/2695 done frame #2500/2695 done frame #2600/2695 done frame #2400/2695 Done anisotropic diffusion #6/22 batch Begin background removal #6/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2100/2695 Done frame #2200/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2400/2695 Done frame #2600/2695 Done background removal #6/22 batch Begin saving Ydebg #6/22 batch Done saving Ydebg #6/22 batch Begin dirts clean #7/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2300/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done dirts clean #7/22 batch Begin anisotropic diffusion #7/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #1500/2695 done frame #500/2695 done frame #800/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #2000/2695 done frame #1800/2695 done frame #2200/2695 done frame #1900/2695 done frame #2100/2695 done frame #1700/2695 done frame #2300/2695 done frame #2400/2695 done frame #2600/2695 done frame #2500/2695 Done anisotropic diffusion #7/22 batch Begin background removal #7/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2500/2695 Done frame #2400/2695 Done frame #2300/2695 Done frame #2600/2695 Done background removal #7/22 batch Begin saving Ydebg #7/22 batch Done saving Ydebg #7/22 batch Begin dirts clean #8/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #2300/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done dirts clean #8/22 batch Begin anisotropic diffusion #8/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #1500/2695 done frame #500/2695 done frame #800/2695 done frame #1400/2695 done frame #1600/2695 done frame #1800/2695 done frame #2000/2695 done frame #1100/2695 done frame #2200/2695 done frame #1900/2695 done frame #1700/2695 done frame #2100/2695 done frame #2300/2695 done frame #2500/2695 done frame #2400/2695 done frame #2600/2695 Done anisotropic diffusion #8/22 batch Begin background removal #8/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2100/2695 Done frame #2400/2695 Done frame #2600/2695 Done frame #2500/2695 Done frame #2300/2695 Done background removal #8/22 batch Begin saving Ydebg #8/22 batch Done saving Ydebg #8/22 batch Begin dirts clean #9/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2200/2695 Done frame #2400/2695 Done frame #2300/2695 Done frame #2600/2695 Done frame #2500/2695 Done dirts clean #9/22 batch Begin anisotropic diffusion #9/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #1500/2695 done frame #500/2695 done frame #800/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #2000/2695 done frame #1800/2695 done frame #2200/2695 done frame #1700/2695 done frame #1900/2695 done frame #2100/2695 done frame #2300/2695 done frame #2600/2695 done frame #2500/2695 done frame #2400/2695 Done anisotropic diffusion #9/22 batch Begin background removal #9/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2300/2695 Done background removal #9/22 batch Begin saving Ydebg #9/22 batch Done saving Ydebg #9/22 batch Begin dirts clean #10/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2600/2695 Done dirts clean #10/22 batch Begin anisotropic diffusion #10/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1500/2695 done frame #500/2695 done frame #1200/2695 done frame #800/2695 done frame #1400/2695 done frame #1600/2695 done frame #1100/2695 done frame #1700/2695 done frame #1800/2695 done frame #2000/2695 done frame #1900/2695 done frame #2200/2695 done frame #2100/2695 done frame #2500/2695 done frame #2400/2695 done frame #2600/2695 done frame #2300/2695 Done anisotropic diffusion #10/22 batch Begin background removal #10/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2200/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2300/2695 Done background removal #10/22 batch Begin saving Ydebg #10/22 batch Done saving Ydebg #10/22 batch Begin dirts clean #11/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2600/2695 Done dirts clean #11/22 batch Begin anisotropic diffusion #11/22 batch done frame #700/2695 done frame #900/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #1200/2695 done frame #1500/2695 done frame #500/2695 done frame #800/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #1800/2695 done frame #2000/2695 done frame #2200/2695 done frame #1900/2695 done frame #1700/2695 done frame #2100/2695 done frame #2300/2695 done frame #2500/2695 done frame #2600/2695 done frame #2400/2695 Done anisotropic diffusion #11/22 batch Begin background removal #11/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2600/2695 Done background removal #11/22 batch Begin saving Ydebg #11/22 batch Done saving Ydebg #11/22 batch Begin dirts clean #12/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2200/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2600/2695 Done frame #2400/2695 Done dirts clean #12/22 batch Begin anisotropic diffusion #12/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1500/2695 done frame #500/2695 done frame #800/2695 done frame #1200/2695 done frame #1100/2695 done frame #1800/2695 done frame #2000/2695 done frame #1600/2695 done frame #2200/2695 done frame #1900/2695 done frame #1700/2695 done frame #1400/2695 done frame #2100/2695 done frame #2500/2695 done frame #2300/2695 done frame #2600/2695 done frame #2400/2695 Done anisotropic diffusion #12/22 batch Begin background removal #12/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2300/2695 Done background removal #12/22 batch Begin saving Ydebg #12/22 batch Done saving Ydebg #12/22 batch Begin dirts clean #13/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2300/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done dirts clean #13/22 batch Begin anisotropic diffusion #13/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1500/2695 done frame #800/2695 done frame #1200/2695 done frame #500/2695 done frame #2000/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #1700/2695 done frame #1800/2695 done frame #2200/2695 done frame #2400/2695 done frame #1900/2695 done frame #2300/2695 done frame #2500/2695 done frame #2600/2695 done frame #2100/2695 Done anisotropic diffusion #13/22 batch Begin background removal #13/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #2500/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2300/2695 Done frame #2600/2695 Done background removal #13/22 batch Begin saving Ydebg #13/22 batch Done saving Ydebg #13/22 batch Begin dirts clean #14/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #2400/2695 Done frame #2600/2695 Done dirts clean #14/22 batch Begin anisotropic diffusion #14/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #1500/2695 done frame #500/2695 done frame #800/2695 done frame #1100/2695 done frame #1600/2695 done frame #1400/2695 done frame #2000/2695 done frame #1800/2695 done frame #2200/2695 done frame #1900/2695 done frame #1700/2695 done frame #2100/2695 done frame #2300/2695 done frame #2500/2695 done frame #2600/2695 done frame #2400/2695 Done anisotropic diffusion #14/22 batch Begin background removal #14/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2500/2695 Done frame #2400/2695 Done frame #2600/2695 Done frame #2300/2695 Done background removal #14/22 batch Begin saving Ydebg #14/22 batch Done saving Ydebg #14/22 batch Begin dirts clean #15/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2300/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done dirts clean #15/22 batch Begin anisotropic diffusion #15/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1500/2695 done frame #500/2695 done frame #1200/2695 done frame #800/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #1800/2695 done frame #2000/2695 done frame #2200/2695 done frame #1900/2695 done frame #1700/2695 done frame #2100/2695 done frame #2300/2695 done frame #2400/2695 done frame #2600/2695 done frame #2500/2695 Done anisotropic diffusion #15/22 batch Begin background removal #15/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2300/2695 Done background removal #15/22 batch Begin saving Ydebg #15/22 batch Done saving Ydebg #15/22 batch Begin dirts clean #16/22 batch Starting parallel pool (parpool) using the 'local' profile ... connected to 32 workers. Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #1700/2695 Done frame #2300/2695 Done frame #2500/2695 Done frame #2400/2695 Done frame #2600/2695 Done dirts clean #16/22 batch Begin anisotropic diffusion #16/22 batch done frame #700/2695 done frame #900/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #500/2695 done frame #1200/2695 done frame #1500/2695 done frame #800/2695 done frame #1800/2695 done frame #1100/2695 done frame #2200/2695 done frame #1700/2695 done frame #2100/2695 done frame #1400/2695 done frame #2400/2695 done frame #1900/2695 done frame #1600/2695 done frame #2600/2695 done frame #2000/2695 done frame #2300/2695 done frame #2500/2695 Done anisotropic diffusion #16/22 batch Begin background removal #16/22 batch Done frame #700/2695 Done frame #1700/2695 Done frame #1600/2695 Done frame #800/2695 Done frame #1100/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #1900/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #2200/2695 Done frame #1300/2695 Done frame #200/2695 Done frame #1500/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1800/2695 Done frame #500/2695 Done frame #600/2695 Done frame #2000/2695 Done frame #2400/2695 Done frame #2100/2695 Done frame #2500/2695 Done frame #1400/2695 Done frame #2300/2695 Done frame #2600/2695 Done background removal #16/22 batch Begin saving Ydebg #16/22 batch Done saving Ydebg #16/22 batch Begin dirts clean #17/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2300/2695 Done frame #2500/2695 Done frame #2400/2695 Done frame #2600/2695 Done dirts clean #17/22 batch Begin anisotropic diffusion #17/22 batch done frame #700/2695 done frame #900/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #400/2695 done frame #600/2695 done frame #300/2695 done frame #500/2695 done frame #1200/2695 done frame #1500/2695 done frame #800/2695 done frame #1800/2695 done frame #2200/2695 done frame #1100/2695 done frame #1600/2695 done frame #1700/2695 done frame #2100/2695 done frame #1400/2695 done frame #1900/2695 done frame #2400/2695 done frame #2300/2695 done frame #2000/2695 done frame #2600/2695 done frame #2500/2695 Done anisotropic diffusion #17/22 batch Begin background removal #17/22 batch Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #1400/2695 Done frame #2200/2695 Done frame #2600/2695 Done frame #2300/2695 Done frame #2500/2695 Done frame #2400/2695 Done background removal #17/22 batch Begin saving Ydebg #17/22 batch Done saving Ydebg #17/22 batch Begin dirts clean #18/22 batch Starting parallel pool (parpool) using the 'local' profile ... connected to 32 workers. Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #2300/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done dirts clean #18/22 batch Begin anisotropic diffusion #18/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #400/2695 done frame #300/2695 done frame #600/2695 done frame #900/2695 done frame #200/2695 done frame #1500/2695 done frame #1200/2695 done frame #800/2695 done frame #500/2695 done frame #1100/2695 done frame #1400/2695 done frame #1600/2695 done frame #2200/2695 done frame #2100/2695 done frame #1800/2695 done frame #2000/2695 done frame #1900/2695 done frame #2300/2695 done frame #1700/2695 done frame #2400/2695 done frame #2600/2695 done frame #2500/2695 Done anisotropic diffusion #18/22 batch Begin background removal #18/22 batch Done frame #1800/2695 Done frame #700/2695 Done frame #800/2695 Done frame #1900/2695 Done frame #900/2695 Done frame #2000/2695 Done frame #1000/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #600/2695 Done frame #2100/2695 Done frame #2300/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2400/2695 Done background removal #18/22 batch Begin saving Ydebg #18/22 batch Done saving Ydebg #18/22 batch Begin dirts clean #19/22 batch Starting parallel pool (parpool) using the 'local' profile ... connected to 32 workers. Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2300/2695 Done frame #2500/2695 Done frame #2400/2695 Done frame #2600/2695 Done dirts clean #19/22 batch Begin anisotropic diffusion #19/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #1500/2695 done frame #800/2695 done frame #1100/2695 done frame #500/2695 done frame #1400/2695 done frame #2200/2695 done frame #1600/2695 done frame #2000/2695 done frame #1900/2695 done frame #2100/2695 done frame #2300/2695 done frame #1700/2695 done frame #2500/2695 done frame #1800/2695 done frame #2600/2695 done frame #2400/2695 Done anisotropic diffusion #19/22 batch Begin background removal #19/22 batch Done frame #700/2695 Done frame #1800/2695 Done frame #800/2695 Done frame #1700/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #600/2695 Done frame #1400/2695 Done frame #2400/2695 Done frame #500/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2300/2695 Done frame #1900/2695 Done frame #2000/2695 Done background removal #19/22 batch Begin saving Ydebg #19/22 batch Done saving Ydebg #19/22 batch Begin dirts clean #20/22 batch Starting parallel pool (parpool) using the 'local' profile ... connected to 32 workers. Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2300/2695 Done frame #2400/2695 Done frame #2500/2695 Done frame #2600/2695 Done dirts clean #20/22 batch Begin anisotropic diffusion #20/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #200/2695 done frame #900/2695 done frame #1200/2695 done frame #1500/2695 done frame #800/2695 done frame #500/2695 done frame #1100/2695 done frame #1400/2695 done frame #2200/2695 done frame #1600/2695 done frame #1700/2695 done frame #2000/2695 done frame #1800/2695 done frame #2100/2695 done frame #1900/2695 done frame #2600/2695 done frame #2400/2695 done frame #2300/2695 done frame #2500/2695 Done anisotropic diffusion #20/22 batch Begin background removal #20/22 batch Done frame #1700/2695 Done frame #700/2695 Done frame #1800/2695 Done frame #900/2695 Done frame #800/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #500/2695 Done frame #600/2695 Done frame #2400/2695 Done frame #2000/2695 Done frame #2500/2695 Done frame #2300/2695 Done frame #1900/2695 Done frame #2200/2695 Done frame #2600/2695 Done background removal #20/22 batch Begin saving Ydebg #20/22 batch Done saving Ydebg #20/22 batch Begin dirts clean #21/22 batch Starting parallel pool (parpool) using the 'local' profile ... connected to 32 workers. Done frame #700/2695 Done frame #1900/2695 Done frame #800/2695 Done frame #2000/2695 Done frame #900/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #1200/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #2200/2695 Done frame #1400/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #1600/2695 Done frame #300/2695 Done frame #400/2695 Done frame #1700/2695 Done frame #500/2695 Done frame #1800/2695 Done frame #600/2695 Done frame #2300/2695 Done frame #2500/2695 Done frame #2400/2695 Done frame #2600/2695 Done dirts clean #21/22 batch Begin anisotropic diffusion #21/22 batch done frame #700/2695 done frame #1000/2695 done frame #100/2695 done frame #1300/2695 done frame #200/2695 done frame #300/2695 done frame #400/2695 done frame #600/2695 done frame #900/2695 done frame #1200/2695 done frame #1500/2695 done frame #500/2695 done frame #800/2695 done frame #1100/2695 done frame #1600/2695 done frame #1400/2695 done frame #2000/2695 done frame #2100/2695 done frame #1900/2695 done frame #1800/2695 done frame #2400/2695 done frame #2300/2695 done frame #2200/2695 done frame #2500/2695 done frame #2600/2695 done frame #1700/2695 Done anisotropic diffusion #21/22 batch Begin background removal #21/22 batch Done frame #1700/2695 Done frame #700/2695 Done frame #800/2695 Done frame #1800/2695 Done frame #900/2695 Done frame #1200/2695 Done frame #1000/2695 Done frame #2100/2695 Done frame #1100/2695 Done frame #100/2695 Done frame #1300/2695 Done frame #1500/2695 Done frame #200/2695 Done frame #300/2695 Done frame #1600/2695 Done frame #400/2695 Done frame #500/2695 Done frame #600/2695 Done frame #2000/2695 Done frame #2400/2695 Done frame #1400/2695 Done frame #2500/2695 Done frame #2600/2695 Done frame #2200/2695 Done frame #2300/2695 Done frame #1900/2695 Done background removal #21/22 batch Begin saving Ydebg #21/22 batch Done saving Ydebg #21/22 batch Begin dirts clean #22/22 batch Starting parallel pool (parpool) using the 'local' profile ... connected to 32 workers. Done frame #700/2678 Done frame #800/2678 Done frame #900/2678 Done frame #2000/2678 Done frame #1000/2678 Done frame #1100/2678 Done frame #2100/2678 Done frame #1200/2678 Done frame #100/2678 Done frame #1300/2678 Done frame #1400/2678 Done frame #200/2678 Done frame #1500/2678 Done frame #300/2678 Done frame #1600/2678 Done frame #400/2678 Done frame #1700/2678 Done frame #500/2678 Done frame #1800/2678 Done frame #600/2678 Done frame #2200/2678 Done frame #2500/2678 Done frame #2300/2678 Done frame #1900/2678 Done frame #2600/2678 Done frame #2400/2678 Done dirts clean #22/22 batch Begin anisotropic diffusion #22/22 batch done frame #900/2678 done frame #100/2678 done frame #400/2678 done frame #1100/2678 done frame #300/2678 done frame #600/2678 done frame #800/2678 done frame #1400/2678 done frame #200/2678 done frame #500/2678 done frame #1000/2678 done frame #1300/2678 done frame #700/2678 done frame #1200/2678 done frame #1500/2678 done frame #1800/2678 done frame #2000/2678 done frame #2200/2678 done frame #1900/2678 done frame #2100/2678 done frame #1600/2678 done frame #1700/2678 done frame #2400/2678 done frame #2600/2678 done frame #2500/2678 done frame #2300/2678 Done anisotropic diffusion #22/22 batch Begin background removal #22/22 batch Done frame #900/2678 Done frame #100/2678 Done frame #400/2678 Done frame #1100/2678 Done frame #800/2678 Done frame #200/2678 Done frame #700/2678 Done frame #1000/2678 Done frame #1300/2678 Done frame #300/2678 Done frame #500/2678 Done frame #600/2678 Done frame #1200/2678 Done frame #1400/2678 Done frame #2000/2678 Done frame #2100/2678 Done frame #1500/2678 Done frame #1900/2678 Done frame #1800/2678 Done frame #2400/2678 Done frame #2300/2678 Done frame #2200/2678 Done frame #2600/2678 Done frame #1700/2678 Done frame #1600/2678 Done frame #2500/2678 Done background removal #22/22 batch Begin saving Ydebg #22/22 batch Done saving Ydebg #22/22 batch Elapsed time is 236397.154091 seconds. Done neural enhancing, time 277442.8037 Starting parallel pool (parpool) using the 'local' profile ... connected to 32 workers. Using GPU Device #1 Total Memory: 11471290368 Begin initial computation of translation score Starting parallel pool (parpool) using the 'local' profile ... connected to 12 workers. computed score of frame #3952/19758 computed score of frame #13832/19758 computed score of frame #5928/19758 computed score of frame #7904/19758 computed score of frame #9880/19758 computed score of frame #1976/19758 computed score of frame #11856/19758 computed score of frame #15808/19758 computed score of frame #17784/19758 Starting parallel pool (parpool) using the 'local' profile ... connected to 12 workers. computed score of frame #3952/19759 computed score of frame #13832/19759 computed score of frame #5928/19759 computed score of frame #7904/19759 computed score of frame #9880/19759 computed score of frame #1976/19759 computed score of frame #11856/19759 computed score of frame #15808/19759 computed score of frame #17784/19759 Starting parallel pool (parpool) using the 'local' profile ... connected to 12 workers. computed score of frame #3952/19758 computed score of frame #13832/19758 computed score of frame #5928/19758 computed score of frame #7904/19758 computed score of frame #9880/19758 computed score of frame #1976/19758 computed score of frame #11856/19758 computed score of frame #15808/19758 computed score of frame #17784/19758 Done initialization, 49732.8718 seconds Begin intra-section Starting parallel pool (parpool) using the 'local' profile ... connected to 12 workers. Done intra_section # 593/5931, batch 1/2 Done intra_section # 1186/5931, batch 1/2 Done intra_section # 1779/5931, batch 1/2 Done intra_section # 2372/5931, batch 1/2 Done intra_section # 2965/5931, batch 1/2 Starting parallel pool (parpool) using the 'local' profile ... connected to 12 workers. Done intra_section # 3559/5931, batch 2/2 Done intra_section # 4152/5931, batch 2/2 Done intra_section # 4745/5931, batch 2/2 Done intra_section # 5338/5931, batch 2/2 Done intra_section # 5931/5931, batch 2/2 Done intra-section, 96671.3199 seconds Begin nonstable-section Starting parallel pool (parpool) using the 'local' profile ... connected to 12 workers. Done nonstable-LogDemons section # 1/2 Done nonstable-LogDemons section # 2/2 Done nonstable-section, 96701.5314 seconds Begin inter-section ... Done data prep Error using gradient_fast (line 9) Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'.

Error in jacobian_matrix2d (line 32) [xdx, xdy] = gradient_fast(vectorField{1});

Error in BCH (line 43) [u_xdx, u_xdy, u_ydx, u_ydy] = jacobian_matrix2d(firstExtraTerm);

Error in logdemons>register (line 161) vcfield = BCH(vfield, ufield);

Error in logdemons (line 78) [Mp, sxl, syl, vxl, vyl] = register(FFl, MMl, opt, isgpu);

Error in logdemons_unit (line 54) [imcurt, sx, sy] = logdemons(demr imref, demr imcur, isgpu, nlevel, sigma_x, sigma_f, sigma_d); %%% with sigma_x = 1 %%%

Error in lk_logdemons_unit (line 93) [img, sxt, syt] = logdemons_unit(imref, imcur, pixs, scl, sigma_x, sigma_f, sigma_f, maskc);

Error in inter_section (line 84) parfor i = 1: nl

Error in frame_reg (line 107) m = inter_section(m, sttn, se, pixs, scl, sigma_x, sigma_f, sigma_d, maskc);

Error in min1pipe (line 103) [m, corr_score, raw_score, scl] = frame_reg(m, imaxy, se, Fsi_new, pixs, scl, sigma_x, sigma_f, sigma_d);

Error in demo_min1pipe (line 14) [fname, frawname, fregname] = min1pipe(Fsi, Fsi_new, spatialr, se, ismc, flag);

IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down. IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down.

JinghaoLu commented 4 years ago

@Les0me The problem is different. The error message indicates out of the GPU memory. To solve this, try using a smaller number of parallel workers (you can change the preferred number of workers in the parallel preferences of matlab). Roughly the #workers/gpu memory <= 4/3.

plodocus commented 4 years ago

If you suggest to decrease the number of workers than this line should also be changed. https://github.com/JinghaoLu/MIN1PIPE/blob/f5740f040c716072587c2181da97e82965778690/utilities/elements/frame_reg.m#L45 Is this necessary at all? By default execution of parallel code initialises a parpool with default settings, so it would be started on demand.

JinghaoLu commented 4 years ago

@DanBenHa You are right. This forces to use the maximum number of workers. Maybe the plain version "parpool" fits more with pre-defined preferred # workers in the parallel preferences. Auto-computing # workers can also be implemented.

plodocus commented 4 years ago

@Les0me's error in https://github.com/JinghaoLu/MIN1PIPE/blob/32084136b3d67cb77f94c1cd3e26d85dc9712f0a/utilities/movement_correction/gradient_fast.m#L9 occurs when gpuArray I is sliced. Apparently, MATLAB makes a memory copy when operating on slices of gpuArrays (I don't know if this is the same behaviour as with other non-GPU arrays), thus temporally exceeding the free memory on the GPU.

I have two questions about gradient_fast().

  1. Why was it added in the first place? MATLAB's inbuilt gradient() gives me slightly better performance.
  2. Shouldn't https://github.com/JinghaoLu/MIN1PIPE/blob/32084136b3d67cb77f94c1cd3e26d85dc9712f0a/utilities/movement_correction/gradient_fast.m#L10 be gx(:, 1) = I(:, 2) - I(:, 1);?

I guess the easiest would be to remove gradient_fast() and replace all its invocations with the inbuilt gradient(). I'm not sure if this would solve the memory issue, though.

Do you want me to make a PR?

JinghaoLu commented 4 years ago

@DanBenHa You are right on the gpuArray slicing issue, when the simultaneously assigned memory by the total active workers exceeds the available GPU memory.

For the gradient_fast(), it is the purified version of the builtin gradient(), and supposed to perform no worse than gradient(). You are right on the second point though.

In general, even if gradient_fast is used, it shouldn't have any impact on the free memory. You are welcome to make a PR for the fix of I(:, 2) - I(:, 1).

marcoistasy commented 4 years ago

So I went from

memo = unix_mem / 2;

to

memo = unix_mem / 4;

But now, it crashes with this error :

Begin inter-section ... Done data prep Done loop #1/6 Done loop #2/6 Done loop #3/6 Done loop #4/6 Done loop #5/6 Done loop #6/6 Unable to perform assignment because the left and right sides have a different number of elements.

Error in inter_section (line 163) idbatch(i) = idtmp;

Error in frame_reg (line 107) m = inter_section(m, sttn, se, pixs, scl, sigma_x, sigma_f, sigma_d, maskc);

Error in min1pipe (line 109) [m, corr_score, raw_score, scl] = frame_reg(m, imaxy, se, Fsi_new, pixs, scl, sigma_x, sigma_f, sigma_d);

IdleTimeout has been reached. Parallel pool using the 'local' profile is shutting down.

Thanks for the help

hi @gviejo and @JinghaoLu! did you ever find a solution to this? I am running into the same problem

JinghaoLu commented 4 years ago

@marcoistasy I cannot replicate the problem, that's why I suggest debugging in matlab. Or you could send the video with the issue to min1pipe2018@gmail.com so that I can see what the problem is.

gviejo commented 4 years ago

Hello sorry for the delay but I have totally shifted project since December. I will not have time to investigate the issue I raised. You can close this issue and open a new one.

Mtkostecki commented 3 years ago

I have an identical error:

Begin inter-section ... Done data prep Error using gradient_fast (line 9) Out of memory.

Error in logdemons>jacobian (line 288) [gy_y, gy_x] = gradient_fast(sy);

Error in logdemons>energy (line 308) jac = jacobian(sx, sy);

Error in logdemons>register (line 173) e(iter) = energy(F, M, sx, sy, opt.sigma_i, opt.sigma_x);

Error in logdemons (line 78) [Mp, sxl, syl, vxl, vyl] = register(FFl, MMl, opt, isgpu);

Error in logdemons_unit (line 59) [imcurt, sx, sy] = logdemons(demr imref, demr imcur, isgpu, nlevel, sigma_x, sigma_f, sigma_d); %%% with sigma_x = 1 %%%

Error in lk_logdemons_unit (line 93) [img, sxt, syt] = logdemons_unit(imref, imcur, pixs, scl, sigma_x, sigma_f, sigma_f, maskc);

Error in inter_section (line 84) parfor i = 1: nl

Error in frame_reg (line 107) m = inter_section(m, sttn, se, pixs, scl, sigma_x, sigma_f, sigma_d, maskc);

Error in min1pipe (line 108) [m, corr_score, raw_score, scl, imaxy] = frame_reg(m, imaxy1, se, Fsi_new, pixs, scl, sigma_x, sigma_f, sigma_d);

Error in demo_min1pipe (line 14) [fname, frawname, fregname] = min1pipe(Fsi, Fsi_new, spatialr, se, ismc, flag);

Should I then decrease the number of workers and change parpool(feature('numCores'))?

JinghaoLu commented 3 years ago

@Mtkostecki I am afraid so. You can calculate the ratio or your number of cpu cores and GPU size. I could not replicate this issue so no idea what exactly happened.