Fusion performance issues

[schmiedc]

Hi Stephan,

About the performance of the weighted-average fusion and the multi view deconvolution.

A) the average-weight fusion using content-based weight takes much longer than in the previous version. In the past it was easily done in the short queue (<1h) now it takes more than that using similar data of similar size. Could you comment on that?

B) the deconvolution (GPU or CPU) is practically not possible on full resolution data. It simply runs out of memory. Is this just the limit of the method or can there be a problem? I tried to use different ImageLib2 containers PlanarImg or CellImg at the fusion also combined with CellImg from the beginning on. But to no avail.

I made the following observations (example data link per Email): B.1) The plugin tells me that I need for example 70GB of memory for the fused image. When it does the fusion it actually uses that much. When it starts the first iteration (0), it goes well beyond until it crashes. Another example with 30GB of necessary memory for the fused imaged need 2-3x as much memory.

B.2) If I use CellImg at the fusion (with or without using CellImg from the beginning) I only get an empty stack (example link per Email)

Cheers, Christopher

[StephanPreibisch]

Hi Christopher,

A) compared to which version? I did not change this code for a long long time. Could be some external factors like Java-version maybe ...

B & C) I re-implemented the deconvolution, many of these problems should be gone. Can you try the new version: http://fly.mpi-cbg.de/preibisch/SPIM_Registration-2.4.0-SNAPSHOT.jar (warning, some parameters are different for the deconvolution. Choose virtual weights first, if you run out of memory still use "save memory")

[schmiedc]

Hi Stephan,

A) sorry, of course. This was compared to the old SPIM Registration plugin, Version 0.55 in the log files. Is this helpful? I think I don't even have that Fiji version anymore. But the difference is striking also to Pavel and Akanksha.

B) thanks a lot! Will test this as soon as possible.

Cheers, Christopher

[StephanPreibisch]

Do you fuse float or unsignedshort?

[schmiedc]

pixel_type=16-bit unsigned integer imglib2_container_fusion=ArrayImg interpolation=Linear Interpolation imglib2_data_container=ArrayImg (faster)

[StephanPreibisch]

try float please ...

[schmiedc]

Good!

[StephanPreibisch]

Means it is faster?

[schmiedc]

Sorry, no just that I will try it. Give me until tomorrow :) Then I give you feedback.

[schmiedc]

First two tests of the deconvolution

1. virtual weights (all other settings preset). Still ran out of memory
2. virtual weights + save memory (all other settings preset) affects my machine somehow badly, I have no access anymore.

But this was with larger than typical dataset which I was successful in deconvolving once at full resolution (really small bounding box), now I cannot reproduce this for the live of me. Will try with one of my own again tomorrow.

[StephanPreibisch]

Can you maybe give me some more details? This should never run out of memory on a small bounding box. What are the exact parameters you ran it with and how much memory do you have available in Java (click on the Fiji status bar).

It makes no sense that it renders you machine useless. Save memory simply re-computes the FFT for the PSF every iteration instead of keeping it. Which Java version are you using?

[schmiedc]

Append to current XML ImgLib2 container: CellImg (large imges ImgLib2 contatiner FFTs: ArrayImg Efficient Baysian - Optimization i (fast precise) Virtual weights (less memory, slower) 1 (balanced) 10 iterations Use Tikhonov Tikhonov parameter 0.0060 Compute 256xz256x256 Compute on CPU (Java) Extract from beds Show MIP

[StephanPreibisch]

Do you also have the bounding box dimensions?

[schmiedc]

Sure: min x: 378 min y: 92 min z: 91 max x: 1606 max y: 1140 max z: 802

[schmiedc]

I am trying now with my samples using the GPU.

[StephanPreibisch]

and the memory available in Fiji?

[schmiedc]

Ah sorry, was set to 124962MB (of 128GB), 24 Parallel threads

[schmiedc]

Now results with GPU different sample. BTW are my libraries still usable with the Snapshot?

Memory set to 124962MB (of 128GB), 24 Parallel threads

Dataset Input dimensions 1TP, 5 views, 1 channel,1 illumination side 1 view: 1200x1920x88px Bounding box: 717x1892x842px offset=275x36x-362px

Set to: Append to current XML ImgLib2 container: CellImg (large imges ImgLib2 contatiner FFTs: ArrayImg Efficient Baysian - Optimization i (fast precise) Virtual weights (less memory, slower) 1 (balanced) 10 iterations Use Tikhonov Tikhonov parameter 0.0060 Compute 256xz256x256 Compute on GPU Extract from beds Show MIP

1. without save memory using virtual weights. Gave me an error out of memory and errors in the console. Sorry forgot to save the log. but this shows that the fusion is done. Systems monitor did not show that it was using too much memory

[schmiedc]

It goes into the first iteration much faster that is for sure.

[StephanPreibisch]

OK, I think I see the problem. You have WAY to many blocks per image (137 at least). Can you please look for the 'effective block size', it is written in one of the logs. Also all the other parameters in the log around that line would be great

[schmiedc]

Sorry for the delay, the machine is freezing on me all the time...

[schmiedc]

Can we exclude that this is some funny CentOS or hardware problem... I can switch to another machine.

[StephanPreibisch]

Could be centos again ... but in any case the Block size of 256x256x256 is way to small relative to the PSF size

[StephanPreibisch]

Also, "save memory" does not apply to the GPU computation, only to CPU computation. There should be absolutely no difference

[schmiedc]

I have actual hardware problems now, network and GPU is down. I will come back to you once this is sorted out.

Ah ok, good to know. So I need to reduce the PSF size in the pop up after the set up.

[StephanPreibisch]

Please check these parameters to be sure: "effective block size", "PSF size", ... please post them here so I can make a reasonable suggestion :)

[schmiedc]

So update,

You were correct. The block size was the solution. I increased it to 512x512x512 now it works. I did not touch the PSF size. CPU deconvolution (something is wrong with the GPU but this is on my side) ) same dataset and boundingbox as before.

Here is the data you asked for:

maxSize: (125, 19, 127) effectiveSizeofblocks(494, 494, 386) Number of blocks: 24 effectiveSizeofblocks(388, 494, 456) Number of blocks: 16 effectiveSizeofblocks(424, 494, 400) Number of blocks: 24 effectiveSizeofblocks(424, 494, 400) Number of blocks: 24 effectiveSizeofblocks(388, 494, 456) Number of blocks: 16

No crashes, stable 1st iteration took 40min

Cheers, Christopher

[StephanPreibisch]

if you can increase the size of the psf even more would be useful, 1024x1024x512 or so

[schmiedc]

Ok, will try that. This is all a bit intransparent for me, maybe I am badly prepared. But is there any way I can read up on this to understand it.

[schmiedc]

Hi Stephan,

Deco run described above ran through. But there was a Problem. The change per pixel is way off. The resulting image looks bad. Blocks missing, different intensities, stripe artifacts. Link for download in Email.

Here is the log file:

Using spimdata version: 0.9-revision Using spimreconstruction version: 2.4.0-SNAPSHOT angles selected: 0, 72, 144, 216, 288, Decon_5 channels selected: Laser 488 illuminations selected: 0, Decon_1 Timepoints selected: 0 Using spimdata version: 0.9-revision Using spimreconstruction version: 2.4.0-SNAPSHOT angles selected: 0, 72, 144, 216, 288 channels selected: Laser 488 illuminations selected: 0 Timepoints selected: 0 Tue Jul 21 21:14:50 CEST 2015: Estimating Bounding Box for Fusion. If size of images is not known (they were never opened before), some of them need to be opened once to determine their size. beads --- channel: Laser 488 angle: 0 illum: 0 timepoint: 0: 419 correspondences. beads --- channel: Laser 488 angle: 72 illum: 0 timepoint: 0: 454 correspondences. beads --- channel: Laser 488 angle: 144 illum: 0 timepoint: 0: 458 correspondences. beads --- channel: Laser 488 angle: 216 illum: 0 timepoint: 0: 468 correspondences. beads --- channel: Laser 488 angle: 288 illum: 0 timepoint: 0: 497 correspondences.

Found 1 label(s) with correspondences for channel Laser 488: Label 'beads' (channel Laser 488) has 5/5 views with corresponding detections. Channel Laser 488: extract PSF from label 'beads' Transforming view 0 of 4 (viewsetup=0, tp=0) (Tue Jul 21 21:15:11 CEST 2015): Reserving memory for transformed & weight image. (Tue Jul 21 21:15:15 CEST 2015): Transformed image factory: CellImgFactory (Tue Jul 21 21:15:15 CEST 2015): Loading image. (Tue Jul 21 21:15:30 CEST 2015): Input image factory: ArrayImgFactory (Tue Jul 21 21:15:30 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Tue Jul 21 21:15:30 CEST 2015): Transforming image & computing weights. (Tue Jul 21 21:15:52 CEST 2015): Extracting PSF for viewsetup 0 using label 'beads' (343 corresponding detections available) PSF size: (19, 19, 25) Transforming view 1 of 4 (viewsetup=1, tp=0) (Tue Jul 21 21:15:54 CEST 2015): Reserving memory for transformed & weight image. (Tue Jul 21 21:15:57 CEST 2015): Transformed image factory: CellImgFactory (Tue Jul 21 21:15:57 CEST 2015): Loading image. (Tue Jul 21 21:16:12 CEST 2015): Input image factory: ArrayImgFactory (Tue Jul 21 21:16:12 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Tue Jul 21 21:16:12 CEST 2015): Transforming image & computing weights. (Tue Jul 21 21:16:45 CEST 2015): Extracting PSF for viewsetup 1 using label 'beads' (371 corresponding detections available) PSF size: (19, 19, 25) Transforming view 2 of 4 (viewsetup=2, tp=0) (Tue Jul 21 21:16:47 CEST 2015): Reserving memory for transformed & weight image. (Tue Jul 21 21:16:50 CEST 2015): Transformed image factory: CellImgFactory (Tue Jul 21 21:16:50 CEST 2015): Loading image. (Tue Jul 21 21:17:05 CEST 2015): Input image factory: ArrayImgFactory (Tue Jul 21 21:17:05 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Tue Jul 21 21:17:05 CEST 2015): Transforming image & computing weights. (Tue Jul 21 21:17:34 CEST 2015): Extracting PSF for viewsetup 2 using label 'beads' (387 corresponding detections available) PSF size: (19, 19, 25) Transforming view 3 of 4 (viewsetup=3, tp=0) (Tue Jul 21 21:17:36 CEST 2015): Reserving memory for transformed & weight image. (Tue Jul 21 21:17:40 CEST 2015): Transformed image factory: CellImgFactory (Tue Jul 21 21:17:40 CEST 2015): Loading image. (Tue Jul 21 21:17:54 CEST 2015): Input image factory: ArrayImgFactory (Tue Jul 21 21:17:54 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Tue Jul 21 21:17:54 CEST 2015): Transforming image & computing weights. (Tue Jul 21 21:18:24 CEST 2015): Extracting PSF for viewsetup 3 using label 'beads' (388 corresponding detections available) PSF size: (19, 19, 25) Transforming view 4 of 4 (viewsetup=4, tp=0) (Tue Jul 21 21:18:25 CEST 2015): Reserving memory for transformed & weight image. (Tue Jul 21 21:18:30 CEST 2015): Transformed image factory: CellImgFactory (Tue Jul 21 21:18:30 CEST 2015): Loading image. (Tue Jul 21 21:18:45 CEST 2015): Input image factory: ArrayImgFactory (Tue Jul 21 21:18:45 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Tue Jul 21 21:18:45 CEST 2015): Transforming image & computing weights. (Tue Jul 21 21:19:17 CEST 2015): Extracting PSF for viewsetup 4 using label 'beads' (396 corresponding detections available) PSF size: (19, 19, 25) (Tue Jul 21 21:19:19 CEST 2015): Computing weight normalization for deconvolution. (Tue Jul 21 21:21:17 CEST 2015): Minimal number of overlapping views: 0, using 1 (Tue Jul 21 21:21:17 CEST 2015): Average number of overlapping views: 2.5252536893113606, using 2.5252536893113606 (Tue Jul 21 21:21:17 CEST 2015): Adjusting for OSEM speedup = 1.0 (Tue Jul 21 21:21:17 CEST 2015): Finished precomputations for deconvolution. Type of iteration: OPTIMIZATION_I Number iterations: 10 OSEM speedup: 1.0 Using blocks: true Block size: (512, 512, 512) Using CUDA: false Blending border: 9x9x5 Blending range: 25x25x25 PSF size (extracting): 19x19x25 Channel Laser 488 extracts from label 'beads'. ImgLib container (deconvolved): CellImgFactory Using Tikhonov regularization (lambda = 0.0060) maxSize: (125, 19, 127) (Tue Jul 21 21:21:17 CEST 2015): Block & FFT image factory: ArrayImgFactory effectiveSize of blocks(494, 494, 386) Number of blocks: 24 effectiveSize of blocks(388, 494, 456) Number of blocks: 16 effectiveSize of blocks(424, 494, 400) Number of blocks: 24 effectiveSize of blocks(424, 494, 400) Number of blocks: 24 effectiveSize of blocks(388, 494, 456) Number of blocks: 16 (Tue Jul 21 21:21:17 CEST 2015): Deconvolved & temporary image factory: CellImgFactory (Tue Jul 21 21:21:21 CEST 2015): Fusing image for first iteration (Tue Jul 21 21:22:40 CEST 2015): Average intensity in overlapping area: 0.006312637065488589 iteration: 0 (Tue Jul 21 21:22:44 CEST 2015) iteration: 0 --- sum change: 9.126773025588667E9 --- max change per pixel: 2.595476E7 iteration: 1 (Tue Jul 21 22:03:55 CEST 2015) iteration: 1 --- sum change: 8.556883717313124E9 --- max change per pixel: 2.4332588E7 iteration: 2 (Tue Jul 21 22:45:07 CEST 2015) iteration: 2 --- sum change: 5.3323305303112763E8 --- max change per pixel: 1520786.75 iteration: 3 (Tue Jul 21 23:25:57 CEST 2015) iteration: 3 --- sum change: 3.3422930705315813E7 --- max change per pixel: 95049.171875 iteration: 4 (Wed Jul 22 00:06:47 CEST 2015) iteration: 4 --- sum change: 2.1941678851233303E10 --- max change per pixel: 5.5417732E7 iteration: 5 (Wed Jul 22 00:47:31 CEST 2015) iteration: 5 --- sum change: 2.057661236542113E10 --- max change per pixel: 5.196904E7 iteration: 6 (Wed Jul 22 01:28:13 CEST 2015) iteration: 6 --- sum change: 1.2800779342191715E9 --- max change per pixel: 3234112.5 iteration: 7 (Wed Jul 22 02:08:59 CEST 2015) iteration: 7 --- sum change: 8.151104588949327E7 --- max change per pixel: 201263.71875 iteration: 8 (Wed Jul 22 02:49:22 CEST 2015) iteration: 8 --- sum change: 3.7488414092570694E10 --- max change per pixel: 7.9395552E7 iteration: 9 (Wed Jul 22 03:30:13 CEST 2015) iteration: 9 --- sum change: 3.5157432972535576E10 --- max change per pixel: 7.4433752E7 DONE (Wed Jul 22 04:11:03 CEST 2015). Wed Jul 22 04:11:42 CEST 2015: Saving file /home/schmied/Desktop/Benchmark/czi/TP0_ChLaser 488_Ill0_Ang0,72,144,216,288.tif Stack is larger than 4GB. Most TIFF readers will only open the first image. Use this information to open as raw: name=Untitled, dir=, width=717, height=1892, nImages=842, offset=8496, type=float, byteOrder=big, format=0, url=, whiteIsZero=f, lutSize=0, comp=1, ranges=null, samples=1 (Wed Jul 22 04:13:03 CEST 2015): Fusion finished. Saved XML '/home/schmied/Desktop/Benchmark/czi/benchmark.xml~2~1'.

[schmiedc]

Hi Stephan, update.

I have tried now 1024x1024x512 (same sample using 2.4.0, CPU, GPU) but it does not work. Fiji crashes all the time.

With the old version 2.3.9 (same sample and bounding box) I arrived now at the following settings Block size 512x512x512, PSF size 10x10x15. Which works both with CPU and GPU on the local machine. I can put the log files here.

I tried to reproduce this with using 2.4.0 using the virtual weights. But this did not work. It says it runs out of memory (>110GB set in Fiji). It is also wired, running the systems monitor side by side did not really show that the processing went to the limit. Additionally one has to restart Fiji, because it indicates that it is still out of memory even though the processing has been aborted. Could this indicate another problem?

Another issues is: I wanted to reproduce the settings in 2.3.9 on the cluster. But this did not work at all both for CPU and GPU. One or two jobs start to run, while the rest were stuck. Although they did run on different nodes. The number of block is maybe too high (16-24), maybe the I/O is the bottle neck in this case, because it needs to copy a lot during the processing.

I will try tomorrow and sunday more.

Cheers, Christopher

[schmiedc]

Here are the log files for the working deconvolution using 2.3.9 using CPU + GPU:

Using spimdata version: 0.9-revision Using spimreconstruction version: 2.3.9 Using spimdata version: 0.9-revision Using spimreconstruction version: 2.3.9 angles selected: 0, 72, 144, 216, 288 channels selected: Laser 488 illuminations selected: 0 Timepoints selected: 0 Thu Jul 23 22:18:36 CEST 2015: Estimating Bounding Box for Fusion. If size of images is not known (they were never opened before), some of them need to be opened once to determine their size. Looking for native libraries ending with '.so', '.lib' in directory: '/home/schmied/Desktop/2015-06-30_Fiji.app.cuda/lib' ... Trying to load following library: /home/schmied/Desktop/2015-06-30_Fiji.app.cuda/lib/libFourierConvolutionCUDALib.so Using an outdated version of the CUDA libs, cannot query free memory. Assuming total memory. Using an outdated version of the CUDA libs, cannot query free memory. Assuming total memory. Using device Quadro 4000 (id=0, mem=2047MB (2047MB free), CUDA capability 2.0) Using device Quadro 4000 (id=1, mem=2047MB (2047MB free), CUDA capability 2.0) beads --- channel: Laser 488 angle: 0 illum: 0 timepoint: 0: 419 correspondences. beads --- channel: Laser 488 angle: 72 illum: 0 timepoint: 0: 454 correspondences. beads --- channel: Laser 488 angle: 144 illum: 0 timepoint: 0: 458 correspondences. beads --- channel: Laser 488 angle: 216 illum: 0 timepoint: 0: 468 correspondences. beads --- channel: Laser 488 angle: 288 illum: 0 timepoint: 0: 497 correspondences.

Found 1 label(s) with correspondences for channel Laser 488: Label 'beads' (channel Laser 488) has 5/5 views with corresponding detections. Thu Jul 23 22:20:00 CEST 2015: Estimating Bounding Box for Fusion. If size of images is not known (they were never opened before), some of them need to be opened once to determine their size. Looking for native libraries ending with '.so', '.lib' in directory: '/home/schmied/Desktop/2015-06-30_Fiji.app.cuda/lib' ... Trying to load following library: /home/schmied/Desktop/2015-06-30_Fiji.app.cuda/lib/libFourierConvolutionCUDALib.so Using an outdated version of the CUDA libs, cannot query free memory. Assuming total memory. Using an outdated version of the CUDA libs, cannot query free memory. Assuming total memory. Using device Quadro 4000 (id=0, mem=2047MB (2047MB free), CUDA capability 2.0) Using device Quadro 4000 (id=1, mem=2047MB (2047MB free), CUDA capability 2.0) beads --- channel: Laser 488 angle: 0 illum: 0 timepoint: 0: 419 correspondences. beads --- channel: Laser 488 angle: 72 illum: 0 timepoint: 0: 454 correspondences. beads --- channel: Laser 488 angle: 144 illum: 0 timepoint: 0: 458 correspondences. beads --- channel: Laser 488 angle: 216 illum: 0 timepoint: 0: 468 correspondences. beads --- channel: Laser 488 angle: 288 illum: 0 timepoint: 0: 497 correspondences.

Found 1 label(s) with correspondences for channel Laser 488: Label 'beads' (channel Laser 488) has 5/5 views with corresponding detections. Channel Laser 488: extract PSF from label 'beads' Fusing view 0 of 4 (Thu Jul 23 22:20:26 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 0 using label 'beads' (343 corresponding detections available) PSF size: (11, 11, 15) Fusing view 1 of 4 (Thu Jul 23 22:21:12 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 1 using label 'beads' (371 corresponding detections available) PSF size: (11, 11, 15) Fusing view 2 of 4 (Thu Jul 23 22:22:13 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 2 using label 'beads' (387 corresponding detections available) PSF size: (11, 11, 15) Fusing view 3 of 4 (Thu Jul 23 22:23:06 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 3 using label 'beads' (388 corresponding detections available) PSF size: (11, 11, 15) Fusing view 4 of 4 (Thu Jul 23 22:24:08 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 4 using label 'beads' (396 corresponding detections available) PSF size: (11, 11, 15) Minimal number of overlapping views: 0, using 1 Average number of overlapping views: 2.5207046984005075, using 2.5207046984005075 Type of iteration: OPTIMIZATION_I Number iterations: 10 OSEM speedup: 1.0 Using blocks: true Block size: (512, 512, 512) Using CUDA: true Blending border: 5x5x3 Blending range: 25x25x25 PSF size (extracting): 11x11x15 Channel Laser 488 extracts from label 'beads'. ImgLib container (deconvolved): ArrayImgFactory Using Tikhonov regularization (lambda = 0.0060) maxSize: (73, 11, 75) Number of blocks: 24 Number of blocks: 16 Number of blocks: 24 Number of blocks: 24 Number of blocks: 16 Thu Jul 23 22:26:33 CEST 2015: numThreads = 24 Thu Jul 23 22:29:31 CEST 2015: done normalizing. Min number of overlapping views: 1 Average number of overlapping views: 2.588066526291904 Average intensity in overlapping area: 0.006879006 OSEM acceleration: 1.0 Deconvolved image container: ArrayContainerFactory iteration: 0 (Thu Jul 23 22:29:38 CEST 2015) iteration: 0 --- sum change: 5621127.651127187 --- max change per pixel: 2.954643487930298 iteration: 1 (Thu Jul 23 22:33:05 CEST 2015) iteration: 1 --- sum change: 1610449.7441699198 --- max change per pixel: 3.761878490447998 iteration: 2 (Thu Jul 23 22:37:35 CEST 2015) iteration: 2 --- sum change: 533070.1448996963 --- max change per pixel: 3.1544899940490723 iteration: 3 (Thu Jul 23 22:43:54 CEST 2015) iteration: 3 --- sum change: 222462.8674598705 --- max change per pixel: 2.681893825531006 iteration: 4 (Thu Jul 23 22:51:52 CEST 2015) iteration: 4 --- sum change: 127827.23620885187 --- max change per pixel: 2.390146255493164 iteration: 5 (Thu Jul 23 22:58:55 CEST 2015) iteration: 5 --- sum change: 96748.6158653262 --- max change per pixel: 2.157878875732422 iteration: 6 (Thu Jul 23 23:05:38 CEST 2015) iteration: 6 --- sum change: 81352.14873379836 --- max change per pixel: 1.9537467956542969 iteration: 7 (Thu Jul 23 23:17:27 CEST 2015) iteration: 7 --- sum change: 70984.8491739363 --- max change per pixel: 1.8273334503173828 iteration: 8 (Thu Jul 23 23:23:50 CEST 2015) iteration: 8 --- sum change: 63283.68896226596 --- max change per pixel: 1.70660400390625 iteration: 9 (Thu Jul 23 23:30:06 CEST 2015) iteration: 9 --- sum change: 57300.22974029028 --- max change per pixel: 1.5830631256103516 DONE (Thu Jul 23 23:40:20 CEST 2015). Thu Jul 23 23:41:21 CEST 2015: Saving file /home/schmied/Desktop/2015-04-21_LZ2_Stock32/TP0_ChLaser 488_Ill0_Ang0,72,144,216,288.tif Stack is larger than 4GB. Most TIFF readers will only open the first image. Use this information to open as raw: name=Untitled, dir=, width=782, height=1911, nImages=904, offset=9116, type=float, byteOrder=big, format=0, url=, whiteIsZero=f, lutSize=0, comp=1, ranges=null, samples=1 (Thu Jul 23 23:43:50 CEST 2015): Fusion finished. Saved XML '/home/schmied/Desktop/2015-04-21_LZ2_Stock32/test1.xml'. Thu Jul 23 23:55:18 CEST 2015: Estimating Bounding Box for Fusion. If size of images is not known (they were never opened before), some of them need to be opened once to determine their size. beads --- channel: Laser 488 angle: 0 illum: 0 timepoint: 0: 419 correspondences. beads --- channel: Laser 488 angle: 72 illum: 0 timepoint: 0: 454 correspondences. beads --- channel: Laser 488 angle: 144 illum: 0 timepoint: 0: 458 correspondences. beads --- channel: Laser 488 angle: 216 illum: 0 timepoint: 0: 468 correspondences. beads --- channel: Laser 488 angle: 288 illum: 0 timepoint: 0: 497 correspondences.

Found 1 label(s) with correspondences for channel Laser 488: Label 'beads' (channel Laser 488) has 5/5 views with corresponding detections. Channel Laser 488: extract PSF from label 'beads' Fusing view 0 of 4 (Thu Jul 23 23:55:26 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 0 using label 'beads' (343 corresponding detections available) PSF size: (11, 11, 15) Fusing view 1 of 4 (Thu Jul 23 23:58:53 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 1 using label 'beads' (371 corresponding detections available) PSF size: (11, 11, 15) Fusing view 2 of 4 (Fri Jul 24 00:00:24 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 2 using label 'beads' (387 corresponding detections available) PSF size: (11, 11, 15) Fusing view 3 of 4 (Fri Jul 24 00:01:24 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 3 using label 'beads' (388 corresponding detections available) PSF size: (11, 11, 15) Fusing view 4 of 4 (Fri Jul 24 00:04:59 CEST 2015): Reserving memory for fused & weight image. Extracting PSF for viewsetup 4 using label 'beads' (396 corresponding detections available) PSF size: (11, 11, 15) Minimal number of overlapping views: 0, using 1 Average number of overlapping views: 2.5207046984005075, using 2.5207046984005075 Type of iteration: OPTIMIZATION_I Number iterations: 10 OSEM speedup: 1.0 Using blocks: true Block size: (512, 512, 512) Using CUDA: false Blending border: 5x5x3 Blending range: 25x25x25 PSF size (extracting): 11x11x15 Channel Laser 488 extracts from label 'beads'. ImgLib container (deconvolved): ArrayImgFactory Using Tikhonov regularization (lambda = 0.0060) maxSize: (73, 11, 75) Number of blocks: 24 Number of blocks: 16 Number of blocks: 24 Number of blocks: 24 Number of blocks: 16 Fri Jul 24 00:11:57 CEST 2015: numThreads = 24 Fri Jul 24 00:15:03 CEST 2015: done normalizing. Min number of overlapping views: 1 Average number of overlapping views: 2.588066526291904 Average intensity in overlapping area: 0.006879006 OSEM acceleration: 1.0 Deconvolved image container: ArrayContainerFactory iteration: 0 (Fri Jul 24 00:15:09 CEST 2015) iteration: 0 --- sum change: 5621127.813598849 --- max change per pixel: 2.9546425342559814 iteration: 1 (Fri Jul 24 00:47:27 CEST 2015) iteration: 1 --- sum change: 1610449.4109396515 --- max change per pixel: 3.7618775367736816 iteration: 2 (Fri Jul 24 01:09:02 CEST 2015) iteration: 2 --- sum change: 533069.9799990326 --- max change per pixel: 3.1544909477233887 iteration: 3 (Fri Jul 24 01:28:41 CEST 2015) iteration: 3 --- sum change: 222462.72282969448 --- max change per pixel: 2.6818933486938477 iteration: 4 (Fri Jul 24 02:12:04 CEST 2015) iteration: 4 --- sum change: 127827.22253166127 --- max change per pixel: 2.3901453018188477 iteration: 5 (Fri Jul 24 02:38:29 CEST 2015) iteration: 5 --- sum change: 96748.60830366914 --- max change per pixel: 2.1578798294067383 iteration: 6 (Fri Jul 24 02:58:57 CEST 2015) iteration: 6 --- sum change: 81352.14973079493 --- max change per pixel: 1.953749656677246 iteration: 7 (Fri Jul 24 03:22:14 CEST 2015) iteration: 7 --- sum change: 70984.84678618405 --- max change per pixel: 1.8273353576660156 iteration: 8 (Fri Jul 24 03:43:11 CEST 2015) iteration: 8 --- sum change: 63283.68868487867 --- max change per pixel: 1.7066020965576172 iteration: 9 (Fri Jul 24 04:07:39 CEST 2015) iteration: 9 --- sum change: 57300.22697267951 --- max change per pixel: 1.5830631256103516 DONE (Fri Jul 24 04:29:24 CEST 2015). Fri Jul 24 04:30:13 CEST 2015: Saving file /home/schmied/Desktop/2015-04-21_LZ2_Stock32/TP0_ChLaser 488_Ill0_Ang0,72,144,216,288.tif Stack is larger than 4GB. Most TIFF readers will only open the first image. Use this information to open as raw: name=Untitled, dir=, width=782, height=1911, nImages=904, offset=9116, type=float, byteOrder=big, format=0, url=, whiteIsZero=f, lutSize=0, comp=1, ranges=null, samples=1 (Fri Jul 24 04:34:31 CEST 2015): Fusion finished. Saved XML '/home/schmied/Desktop/2015-04-21_LZ2_Stock32/test1.xml'.

[schmiedc]

Here is the log file for the deconvolution 2.4.0 virtual weights same settings GPU:

Using spimdata version: 0.9-revision Using spimreconstruction version: 2.4.0 angles selected: 0, 72, 144, 216, 288 channels selected: 488 illuminations selected: 0 Timepoints selected: 0 Sat Jul 25 01:12:49 CEST 2015: Estimating Bounding Box for Fusion. If size of images is not known (they were never opened before), some of them need to be opened once to determine their size. Looking for native libraries ending with '.so', '.lib' in directory: '/home/schmied/Desktop/2015-07-24_Fiji.app.cuda/lib' ... Trying to load following library: /home/schmied/Desktop/2015-07-24_Fiji.app.cuda/lib/libFourierConvolutionCUDALib.so Using an outdated version of the CUDA libs, cannot query free memory. Assuming total memory. Using an outdated version of the CUDA libs, cannot query free memory. Assuming total memory. Using device Quadro 4000 (id=0, mem=2047MB (2047MB free), CUDA capability 2.0) Using device Quadro 4000 (id=1, mem=2047MB (2047MB free), CUDA capability 2.0) beads --- channel: 488 angle: 0 illum: 0 timepoint: 0: 419 correspondences. beads --- channel: 488 angle: 72 illum: 0 timepoint: 0: 454 correspondences. beads --- channel: 488 angle: 144 illum: 0 timepoint: 0: 458 correspondences. beads --- channel: 488 angle: 216 illum: 0 timepoint: 0: 468 correspondences. beads --- channel: 488 angle: 288 illum: 0 timepoint: 0: 497 correspondences.

Found 1 label(s) with correspondences for channel 488: Label 'beads' (channel 488) has 5/5 views with corresponding detections. Channel 488: extract PSF from label 'beads' Transforming view 0 of 4 (viewsetup=0, tp=0) (Sat Jul 25 01:13:40 CEST 2015): Reserving memory for transformed & weight image. (Sat Jul 25 01:13:45 CEST 2015): Transformed image factory: CellImgFactory (Sat Jul 25 01:13:45 CEST 2015): Loading image. (Sat Jul 25 01:13:54 CEST 2015): Input image factory: ArrayImgFactory (Sat Jul 25 01:13:54 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Sat Jul 25 01:13:54 CEST 2015): Transforming image & computing weights. (Sat Jul 25 01:14:18 CEST 2015): Extracting PSF for viewsetup 0 using label 'beads' (343 corresponding detections available) PSF size: (11, 11, 15) Transforming view 1 of 4 (viewsetup=1, tp=0) (Sat Jul 25 01:14:19 CEST 2015): Reserving memory for transformed & weight image. (Sat Jul 25 01:14:23 CEST 2015): Transformed image factory: CellImgFactory (Sat Jul 25 01:14:23 CEST 2015): Loading image. (Sat Jul 25 01:14:31 CEST 2015): Input image factory: ArrayImgFactory (Sat Jul 25 01:14:31 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Sat Jul 25 01:14:31 CEST 2015): Transforming image & computing weights. (Sat Jul 25 01:15:03 CEST 2015): Extracting PSF for viewsetup 1 using label 'beads' (371 corresponding detections available) PSF size: (11, 11, 15) Transforming view 2 of 4 (viewsetup=2, tp=0) (Sat Jul 25 01:15:03 CEST 2015): Reserving memory for transformed & weight image. (Sat Jul 25 01:15:08 CEST 2015): Transformed image factory: CellImgFactory (Sat Jul 25 01:15:08 CEST 2015): Loading image. (Sat Jul 25 01:15:17 CEST 2015): Input image factory: ArrayImgFactory (Sat Jul 25 01:15:17 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Sat Jul 25 01:15:17 CEST 2015): Transforming image & computing weights. (Sat Jul 25 01:15:49 CEST 2015): Extracting PSF for viewsetup 2 using label 'beads' (387 corresponding detections available) PSF size: (11, 11, 15) Transforming view 3 of 4 (viewsetup=3, tp=0) (Sat Jul 25 01:15:49 CEST 2015): Reserving memory for transformed & weight image. (Sat Jul 25 01:15:56 CEST 2015): Transformed image factory: CellImgFactory (Sat Jul 25 01:15:56 CEST 2015): Loading image. (Sat Jul 25 01:16:05 CEST 2015): Input image factory: ArrayImgFactory (Sat Jul 25 01:16:05 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Sat Jul 25 01:16:05 CEST 2015): Transforming image & computing weights. (Sat Jul 25 01:16:39 CEST 2015): Extracting PSF for viewsetup 3 using label 'beads' (388 corresponding detections available) PSF size: (11, 11, 15) Transforming view 4 of 4 (viewsetup=4, tp=0) (Sat Jul 25 01:16:40 CEST 2015): Reserving memory for transformed & weight image. (Sat Jul 25 01:16:47 CEST 2015): Transformed image factory: CellImgFactory (Sat Jul 25 01:16:47 CEST 2015): Loading image. (Sat Jul 25 01:16:55 CEST 2015): Input image factory: ArrayImgFactory (Sat Jul 25 01:16:55 CEST 2015): Initializing transformation & weights: VIRTUAL_WEIGHTS (Sat Jul 25 01:16:55 CEST 2015): Transforming image & computing weights. (Sat Jul 25 01:17:32 CEST 2015): Extracting PSF for viewsetup 4 using label 'beads' (396 corresponding detections available) PSF size: (11, 11, 15) (Sat Jul 25 01:17:32 CEST 2015): Computing weight normalization for deconvolution. (Sat Jul 25 01:19:45 CEST 2015): Minimal number of overlapping views: 0, using 1 (Sat Jul 25 01:19:45 CEST 2015): Average number of overlapping views: 2.519386370008137, using 2.519386370008137 (Sat Jul 25 01:19:45 CEST 2015): Adjusting for OSEM speedup = 1.0 (Sat Jul 25 01:19:45 CEST 2015): Finished precomputations for deconvolution. Type of iteration: OPTIMIZATION_I Number iterations: 10 OSEM speedup: 1.0 Using blocks: true Block size: (512, 512, 512) Using CUDA: true Blending border: 5x5x3 Blending range: 25x25x25 PSF size (extracting): 11x11x15 Channel 488 extracts from label 'beads'. ImgLib container (deconvolved): CellImgFactory Using Tikhonov regularization (lambda = 0.0060) maxSize: (73, 11, 75) (Sat Jul 25 01:19:46 CEST 2015): Block & FFT image factory: ArrayImgFactory effectiveSize of blocks(502, 502, 438) Number of blocks: 24 effectiveSize of blocks(440, 502, 480) Number of blocks: 16 effectiveSize of blocks(460, 502, 446) Number of blocks: 24 effectiveSize of blocks(462, 502, 448) Number of blocks: 24 effectiveSize of blocks(440, 502, 480) Number of blocks: 16 (Sat Jul 25 01:19:46 CEST 2015): Deconvolved & temporary image factory: CellImgFactory (Sat Jul 25 01:19:47 CEST 2015): Fusing image for first iteration (Sat Jul 25 01:21:16 CEST 2015): Average intensity in overlapping area: 0.005580830968034308 iteration: 0 (Sat Jul 25 01:21:22 CEST 2015) Out of Memory (Sat Jul 25 01:24:28 CEST 2015): Fusion finished.

[schmiedc]

Add 2.4.0 virtual weights same settings GPU: print out of console

Sorry for the potato quality. But the screenshots stopped working (out of memory).

[lacan]

I can imagine that this thread was fixed since two years ago, but to what conclusion did you arrive?

Best

Oli

[schmiedc]

Hi I don't know how applicable my experiences from 2015 still are ;) So take it with two grains of salt. I think I could not make the GPU deconvolution work with the at that time most updated version. Switched to CPU then. But I remember vaguely that this could be related to a wrong cuda version.

Against crashes I did: Downsampling at least by a factor of 2. But that I guess depends on how large your images are. Then I kept the PSF size as small as possible (Could reduce quality of deconvolution thought) For blocks I used 512x512x512 not smaller. PSF size and block size determine the number of blocks and keeping that number small seemed to prevent these kind of crashes.

For GPU in the past downsampling was crucial. But this is also maybe related to the memory of your GPU and image size.

Cheers,

Christopher