ptycho / ptypy

Ptypy - main repository
Other
35 stars 15 forks source link

Recontructed object larger than 50Mb issue #511

Closed ltang320 closed 7 months ago

ltang320 commented 11 months ago

In our reconstruction, we want a large object under 10nm pixel size. However, the object size will be larger than 50Mb. In this case, it will raise the error that arrays larger than 50M are unsupported. Is there any way to fix it? @daurer @bjoernenders

`
raise RuntimeError('Arrays larger than %dM not supported. You ' RuntimeError: Arrays larger than 50M not supported. You requested 193.99M pixels. raise RuntimeError('Arrays larger than %dM not supported. You ' RuntimeError: Arrays larger than 50M not supported. You requested 193.99M pixels. raise RuntimeError('Arrays larger than %dM not supported. You ' RuntimeError: Arrays larger than 50M not supported. You requested 193.99M pixels. raise RuntimeError('Arrays larger than %dM not supported. You ' RuntimeError: Arrays larger than 50M not supported. You requested 193.99M pixels.

Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted.

mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was:

Process name: [[34300,1],3]
Exit code:    1

`

daurer commented 11 months ago

Hi, Sure, we can increase that limit if needed. However, I am wondering if you are indeed running into this limit or if perhaps there might be something else wrong in your geometry.

The error message suggests that ptypy is trying to create an object with 194 Million pixels, which equals to an array of 13928388x13928388 pixels. At 10 nm pixel size, this would give you a field of view of 139 mm - this seems rather large, does it make sense?

Again, happy to make a change to extend the limit but I would double check your other parameters, i.e. detector pixel size, photon energy and scan positions and make sure they are provided with the correct units.

ltang320 commented 11 months ago

Hi, Sure, we can increase that limit if needed. However, I am wondering if you are indeed running into this limit or if perhaps there might be something else wrong in your geometry.

The error message suggests that ptypy is trying to create an object with 194 Million pixels, which equals to an array of 13928388x13928388 pixels. At 10 nm pixel size, this would give you a field of view of 139 mm - this seems rather large, does it make sense?

Again, happy to make a change to extend the limit but I would double check your other parameters, i.e. detector pixel size, photon energy and scan positions and make sure they are provided with the correct units.

In our reconstruction, the object size is 131.96um x 103um and our pixel size is 8.2nm. In this case, the array should be about 16092 pix x 12652 pix (~203 Million pixels). In this case, the object size is about 192Mb. I don't think these is anything wrong in this calculation. I guess here you calculate the FOV wrong. It should be ~ 139um.

If you can make this change, it would be very useful for us. Could you let me know when will this change be done? Thanks a lot for your help!

daurer commented 10 months ago

@ltang320 sorry, my mistake! Indeed, I got the calculation of the FOV wrong - it should have been 13928x13928 and 139 x 139 um which is still quite large I guess but sounds more reasonable.

I have made a quick change on the megapixel-limit branch (#513 ) and raised the limit to 500. Can you check if that works for you?

ltang320 commented 10 months ago

@ltang320 sorry, my mistake! Indeed, I got the calculation of the FOV wrong - it should have been 13928x13928 and 139 x 139 um which is still quite large I guess but sounds more reasonable.

I have made a quick change on the megapixel-limit branch (#513 ) and raised the limit to 500. Can you check if that works for you?

@daurer I checked it. And I end up with this memory issue. Here is the runtime issue.

P.run() File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 784, in run self.run(engine=engine) File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 662, in run engine.initialize() File "/home/litang/ptypy_megapix/build/lib/ptypy/engines/base.py", line 154, in initialize self.engine_initialize() File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda_stream.py", line 49, in engine_initialize super().engine_initialize() File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda.py", line 95, in engine_initialize super().engine_initialize() File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 137, in engine_initialize self._setup_kernels() File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda_stream.py", line 55, in _setup_kernels super()._setup_kernels() File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda.py", line 146, in _setup_kernels kern.PROP.allocate() File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/kernels.py", line 69, in allocate self._fft2 = FFT(self._tmp, self.queue, ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/fft.py", line 79, in init self._set_stream(thr) File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/fft.py", line 94, in _set_stream self._ftreikna = self._ftreikna_raw.compile(thr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/core/computation.py", line 207, in compile self._tr_tree, translator, thread, fast_math, compiler_options, keep).finalize() ^^^^^^^^^^ File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/core/computation.py", line 557, in finalize new_buf = self._thread.temp_array( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/api.py", line 403, in temp_array return self.temp_alloc.array( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/tempalloc.py", line 79, in array self._allocate(new_id, array.nbytes, dependencies, self._pack_on_alloc) File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/tempalloc.py", line 181, in _allocate self._fast_add(new_id, size, dep_set) File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/tempalloc.py", line 205, in _fast_add buf = self._thr.allocate(size) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/cuda.py", line 168, in allocate return Buffer(size) ^^^^^^^^^^^^ File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/cuda.py", line 60, in init self._buffer = cuda.mem_alloc(size) ^^^^^^^^^^^^^^^^^^^^ pycuda._driver.MemoryError: cuMemAlloc failed: out of memory Exception ignored in: <function Buffer.del at 0x2ac61913dd00> Traceback (most recent call last): File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/cuda.py", line 70, in del self._buffer.free() ^^^^^^^^^^^^ AttributeError: 'Buffer' object has no attribute '_buffer' Traceback (most recent call last): File "/asap3/petra3/gpfs/p06/2023/data/11016011/processed/macros/rec_multi_3x4_GPU_init_P_TL_crop512.py", line 352, in P.run() File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 784, in run self.run(engine=engine) File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 669, in run engine.prepare() File "/home/litang/ptypy_megapix/build/lib/ptypy/engines/base.py", line 182, in prepare self.engine_prepare() File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda_stream.py", line 83, in engine_prepare super(projectional_pycuda._ProjectionEngine_pycuda, self).engine_prepare() File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 223, in engine_prepare prep.view_IDs, prep.poe_IDs, prep.addr = serialize_array_access(d) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 59, in serialize_array_access mpod = views[0].pod


IndexError: list index out of range
Traceback (most recent call last):
  File "/asap3/petra3/gpfs/p06/2023/data/11016011/processed/macros/rec_multi_3x4_GPU_init_P_TL_crop512.py", line 352, in <module>
    P.run()
  File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 784, in run
    self.run(engine=engine)
  File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 669, in run
    engine.prepare()
  File "/home/litang/ptypy_megapix/build/lib/ptypy/engines/base.py", line 182, in prepare
    self.engine_prepare()
  File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda_stream.py", line 83, in engine_prepare
    super(projectional_pycuda._ProjectionEngine_pycuda, self).engine_prepare()
  File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 223, in engine_prepare
    prep.view_IDs, prep.poe_IDs, prep.addr = serialize_array_access(d)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 59, in serialize_array_access
    mpod = views[0].pod
           ~~~~~^^^
IndexError: list index out of range
Traceback (most recent call last):
  File "/asap3/petra3/gpfs/p06/2023/data/11016011/processed/macros/rec_multi_3x4_GPU_init_P_TL_crop512.py", line 352, in <module>
    P.run()
  File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 784, in run
    self.run(engine=engine)
  File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 669, in run
    engine.prepare()
  File "/home/litang/ptypy_megapix/build/lib/ptypy/engines/base.py", line 182, in prepare
    self.engine_prepare()
  File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda_stream.py", line 83, in engine_prepare
    super(projectional_pycuda._ProjectionEngine_pycuda, self).engine_prepare()
  File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 223, in engine_prepare
    prep.view_IDs, prep.poe_IDs, prep.addr = serialize_array_access(d)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 59, in serialize_array_access
    mpod = views[0].pod
           ~~~~~^^^
IndexError: list index out of range
    -------------------------------------------------------------------
    PyCUDA ERROR: The context stack was not empty upon module cleanup.
    -------------------------------------------------------------------
    A context was still active when the context stack was being
    cleaned up. At this point in our execution, CUDA may already
    have been deinitialized, so there is no way we can finish
    cleanly. The program will be aborted now.
    Use Context.pop() to avoid this problem.
    -------------------------------------------------------------------
daurer commented 10 months ago

@ltang320 looks like you are running out of GPU memory. Whats the size of your dataset, how many frames and how large and what do you use as frames_per_block in the run script?

ltang320 commented 10 months ago

@ltang320 looks like you are running out of GPU memory. Whats the size of your dataset, how many frames and how large and what do you use as frames_per_block in the run script?

@daurer There are 10201 frames in this dataset. I tried 25 frames_per_block. From the memory taken for each pods, it doesn't seem that it can be over the memory. And I am using 4 core 80GB GPU now.

---- Ptycho init level 3 -------------------------------------------------------

Process #3 ---- Total Pods 122412 (30612 active) ----

(C)ontnr : Memory : Shape : Pixel size : Dimensions : Views (S)torgs : (MB) : (Pixel) : (meters) : (meters) : act.

Cprobe : 25.2 : complex64 Sscan00G00 : 25.2 : 12 512 512 : 8.18.1e-9 : 4.24.2e-6 : 22412 Cobj : 186.0 : complex64 Sscan00G00 : 186.0 : 1 4819 4825 : 8.18.1e-9 : 3.93.9e-5 : 22412 Cexit : 64198.0 : complex64 S0000G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0001G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0002G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0003G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0004G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0005G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0006G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0007G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0008G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0009G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0010G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0011G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0012G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0013G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0014G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0015G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0016G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0017G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0018G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0019G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0020G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0021G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0022G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0023G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0024G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0025G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0026G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0027G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0028G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0029G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0030G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0031G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0032G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0033G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0034G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0035G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0036G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0037G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0038G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0039G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0040G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0041G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0042G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0043G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0044G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0045G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0046G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0047G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0048G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0049G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0050G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0051G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0052G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0053G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0054G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0055G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0056G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0057G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0058G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0059G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0060G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0061G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0062G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0063G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0064G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0065G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0066G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0067G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0068G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0069G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0070G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0071G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0072G00 : 780.1 : 372 512 512 : 8.18.1e-9 : 4.24.2e-6 : 372 Cdiff : 2674.9 : float32 S0000 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0001 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0002 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0003 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0004 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0005 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0006 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0007 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0008 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0009 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0010 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0011 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0012 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0013 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0014 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0015 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0016 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0017 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0018 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0019 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0020 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0021 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0022 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0023 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0024 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0025 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0026 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0027 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0028 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0029 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0030 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0031 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0032 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0033 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0034 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0035 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0036 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0037 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0038 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0039 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0040 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0041 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0042 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0043 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0044 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0045 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0046 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0047 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0048 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0049 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0050 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0051 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0052 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0053 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0054 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0055 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0056 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0057 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0058 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0059 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0060 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0061 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0062 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0063 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0064 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0065 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0066 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0067 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0068 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0069 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0070 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0071 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0072 : 32.5 : 31 512 512 : 7.57.5e-5 : 3.83.8e-2 : 31 Cmask : 668.7 : bool S0000 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0001 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0002 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0003 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0004 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0005 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0006 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0007 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0008 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0009 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0010 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0011 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0012 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0013 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0014 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0015 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0016 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0017 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0018 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0019 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0020 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0021 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0022 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0023 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0024 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0025 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0026 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0027 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0028 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0029 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0030 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0031 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0032 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0033 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0034 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0035 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0036 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0037 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0038 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0039 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0040 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0041 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0042 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0043 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0044 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0045 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0046 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0047 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0048 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0049 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0050 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0051 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0052 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0053 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0054 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0055 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0056 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0057 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0058 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0059 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0060 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0061 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0062 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0063 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0064 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0065 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0066 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0067 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0068 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0069 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0070 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0071 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0072 : 8.1 : 31 512 512 : 7.57.5e-5 : 3.83.8e-2 : 31

---- Ptycho init level 4 ------------------------------------------------------- object size before (1, 4819, 4825) object size before (1, 4819, 4825) object size before (1, 4819, 4825) object size before (1, 4819, 4825) object size after (1, 12181, 15926)

==== Starting DM_pycuda-algorithm. =============================================

Parameter set:

daurer commented 10 months ago

Thanks for the info. The memory needed to run DM on the GPU is not just the pods, there are some other auxilliary arrays needed that scale with the nr. of modes and the frames per block. It looks like you are using 12 modes in this example and from what is shown above it looks like 35 frames per block were used.

Unfortunately, our internal memory calculation are not that great at the moment that's why you might just this ugly pycuda traceback whenever you run out of memory on the GPU. If it keeps happening, the only thing I can recommend is lowering the frames per block, lowering the modes or increasing the GPU memory, i.e. by scaling the task over multiple GPUs/nodes.

daurer commented 7 months ago

The original issue (megapixel limit) has been fixed with release 0.8. If there are still remaining memory issue for this use case, feel free to open a new issue.