Closed ltang320 closed 9 months ago
Hi, Sure, we can increase that limit if needed. However, I am wondering if you are indeed running into this limit or if perhaps there might be something else wrong in your geometry.
The error message suggests that ptypy is trying to create an object with 194 Million pixels, which equals to an array of 13928388x13928388 pixels. At 10 nm pixel size, this would give you a field of view of 139 mm - this seems rather large, does it make sense?
Again, happy to make a change to extend the limit but I would double check your other parameters, i.e. detector pixel size, photon energy and scan positions and make sure they are provided with the correct units.
Hi, Sure, we can increase that limit if needed. However, I am wondering if you are indeed running into this limit or if perhaps there might be something else wrong in your geometry.
The error message suggests that ptypy is trying to create an object with 194 Million pixels, which equals to an array of 13928388x13928388 pixels. At 10 nm pixel size, this would give you a field of view of 139 mm - this seems rather large, does it make sense?
Again, happy to make a change to extend the limit but I would double check your other parameters, i.e. detector pixel size, photon energy and scan positions and make sure they are provided with the correct units.
In our reconstruction, the object size is 131.96um x 103um and our pixel size is 8.2nm. In this case, the array should be about 16092 pix x 12652 pix (~203 Million pixels). In this case, the object size is about 192Mb. I don't think these is anything wrong in this calculation. I guess here you calculate the FOV wrong. It should be ~ 139um.
If you can make this change, it would be very useful for us. Could you let me know when will this change be done? Thanks a lot for your help!
@ltang320 sorry, my mistake! Indeed, I got the calculation of the FOV wrong - it should have been 13928x13928 and 139 x 139 um which is still quite large I guess but sounds more reasonable.
I have made a quick change on the megapixel-limit
branch (#513 ) and raised the limit to 500. Can you check if that works for you?
@ltang320 sorry, my mistake! Indeed, I got the calculation of the FOV wrong - it should have been 13928x13928 and 139 x 139 um which is still quite large I guess but sounds more reasonable.
I have made a quick change on the
megapixel-limit
branch (#513 ) and raised the limit to 500. Can you check if that works for you?
@daurer I checked it. And I end up with this memory issue. Here is the runtime issue.
P.run()
File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 784, in run
self.run(engine=engine)
File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 662, in run
engine.initialize()
File "/home/litang/ptypy_megapix/build/lib/ptypy/engines/base.py", line 154, in initialize
self.engine_initialize()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda_stream.py", line 49, in engine_initialize
super().engine_initialize()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda.py", line 95, in engine_initialize
super().engine_initialize()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 137, in engine_initialize
self._setup_kernels()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda_stream.py", line 55, in _setup_kernels
super()._setup_kernels()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda.py", line 146, in _setup_kernels
kern.PROP.allocate()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/kernels.py", line 69, in allocate
self._fft2 = FFT(self._tmp, self.queue,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/fft.py", line 79, in init
self._set_stream(thr)
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/fft.py", line 94, in _set_stream
self._ftreikna = self._ftreikna_raw.compile(thr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/core/computation.py", line 207, in compile
self._tr_tree, translator, thread, fast_math, compiler_options, keep).finalize()
^^^^^^^^^^
File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/core/computation.py", line 557, in finalize
new_buf = self._thread.temp_array(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/api.py", line 403, in temp_array
return self.temp_alloc.array(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/tempalloc.py", line 79, in array
self._allocate(new_id, array.nbytes, dependencies, self._pack_on_alloc)
File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/tempalloc.py", line 181, in _allocate
self._fast_add(new_id, size, dep_set)
File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/tempalloc.py", line 205, in _fast_add
buf = self._thr.allocate(size)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/cuda.py", line 168, in allocate
return Buffer(size)
^^^^^^^^^^^^
File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/cuda.py", line 60, in init
self._buffer = cuda.mem_alloc(size)
^^^^^^^^^^^^^^^^^^^^
pycuda._driver.MemoryError: cuMemAlloc failed: out of memory
Exception ignored in: <function Buffer.del at 0x2ac61913dd00>
Traceback (most recent call last):
File "/home/litang/.conda/envs/env_ptypy_megapix/lib/python3.11/site-packages/reikna/cluda/cuda.py", line 70, in del
self._buffer.free()
^^^^^^^^^^^^
AttributeError: 'Buffer' object has no attribute '_buffer'
Traceback (most recent call last):
File "/asap3/petra3/gpfs/p06/2023/data/11016011/processed/macros/rec_multi_3x4_GPU_init_P_TL_crop512.py", line 352, in
IndexError: list index out of range
Traceback (most recent call last):
File "/asap3/petra3/gpfs/p06/2023/data/11016011/processed/macros/rec_multi_3x4_GPU_init_P_TL_crop512.py", line 352, in <module>
P.run()
File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 784, in run
self.run(engine=engine)
File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 669, in run
engine.prepare()
File "/home/litang/ptypy_megapix/build/lib/ptypy/engines/base.py", line 182, in prepare
self.engine_prepare()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda_stream.py", line 83, in engine_prepare
super(projectional_pycuda._ProjectionEngine_pycuda, self).engine_prepare()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 223, in engine_prepare
prep.view_IDs, prep.poe_IDs, prep.addr = serialize_array_access(d)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 59, in serialize_array_access
mpod = views[0].pod
~~~~~^^^
IndexError: list index out of range
Traceback (most recent call last):
File "/asap3/petra3/gpfs/p06/2023/data/11016011/processed/macros/rec_multi_3x4_GPU_init_P_TL_crop512.py", line 352, in <module>
P.run()
File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 784, in run
self.run(engine=engine)
File "/home/litang/ptypy_megapix/build/lib/ptypy/core/ptycho.py", line 669, in run
engine.prepare()
File "/home/litang/ptypy_megapix/build/lib/ptypy/engines/base.py", line 182, in prepare
self.engine_prepare()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/cuda_pycuda/engines/projectional_pycuda_stream.py", line 83, in engine_prepare
super(projectional_pycuda._ProjectionEngine_pycuda, self).engine_prepare()
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 223, in engine_prepare
prep.view_IDs, prep.poe_IDs, prep.addr = serialize_array_access(d)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/litang/ptypy_megapix/build/lib/ptypy/accelerate/base/engines/projectional_serial.py", line 59, in serialize_array_access
mpod = views[0].pod
~~~~~^^^
IndexError: list index out of range
-------------------------------------------------------------------
PyCUDA ERROR: The context stack was not empty upon module cleanup.
-------------------------------------------------------------------
A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.
-------------------------------------------------------------------
@ltang320 looks like you are running out of GPU memory. Whats the size of your dataset, how many frames and how large and what do you use as frames_per_block
in the run script?
@ltang320 looks like you are running out of GPU memory. Whats the size of your dataset, how many frames and how large and what do you use as
frames_per_block
in the run script?
@daurer There are 10201 frames in this dataset. I tried 25 frames_per_block. From the memory taken for each pods, it doesn't seem that it can be over the memory. And I am using 4 core 80GB GPU now.
---- Ptycho init level 3 -------------------------------------------------------
Cprobe : 25.2 : complex64 Sscan00G00 : 25.2 : 12 512 512 : 8.18.1e-9 : 4.24.2e-6 : 22412 Cobj : 186.0 : complex64 Sscan00G00 : 186.0 : 1 4819 4825 : 8.18.1e-9 : 3.93.9e-5 : 22412 Cexit : 64198.0 : complex64 S0000G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0001G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0002G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0003G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0004G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0005G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0006G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0007G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0008G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0009G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0010G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0011G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0012G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0013G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0014G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0015G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0016G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0017G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0018G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0019G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0020G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0021G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0022G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0023G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0024G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0025G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0026G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0027G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0028G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0029G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0030G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0031G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0032G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0033G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0034G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0035G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0036G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0037G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0038G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0039G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0040G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0041G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0042G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0043G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0044G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0045G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0046G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0047G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0048G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0049G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0050G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0051G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0052G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0053G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0054G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0055G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0056G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0057G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0058G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0059G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0060G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0061G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0062G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0063G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0064G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0065G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0066G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0067G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0068G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0069G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0070G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0071G00 : 880.8 : 420 512 512 : 8.18.1e-9 : 4.24.2e-6 : 420 S0072G00 : 780.1 : 372 512 512 : 8.18.1e-9 : 4.24.2e-6 : 372 Cdiff : 2674.9 : float32 S0000 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0001 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0002 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0003 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0004 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0005 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0006 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0007 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0008 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0009 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0010 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0011 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0012 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0013 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0014 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0015 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0016 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0017 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0018 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0019 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0020 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0021 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0022 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0023 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0024 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0025 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0026 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0027 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0028 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0029 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0030 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0031 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0032 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0033 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0034 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0035 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0036 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0037 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0038 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0039 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0040 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0041 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0042 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0043 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0044 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0045 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0046 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0047 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0048 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0049 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0050 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0051 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0052 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0053 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0054 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0055 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0056 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0057 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0058 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0059 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0060 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0061 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0062 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0063 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0064 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0065 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0066 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0067 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0068 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0069 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0070 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0071 : 36.7 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0072 : 32.5 : 31 512 512 : 7.57.5e-5 : 3.83.8e-2 : 31 Cmask : 668.7 : bool S0000 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0001 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0002 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0003 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0004 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0005 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0006 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0007 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0008 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0009 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0010 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0011 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0012 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0013 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0014 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0015 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0016 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0017 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0018 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0019 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0020 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0021 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0022 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0023 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0024 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0025 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0026 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0027 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0028 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0029 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0030 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0031 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0032 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0033 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0034 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0035 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0036 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0037 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0038 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0039 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0040 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0041 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0042 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0043 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0044 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0045 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0046 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0047 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0048 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0049 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0050 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0051 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0052 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0053 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0054 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0055 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0056 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0057 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0058 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0059 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0060 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0061 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0062 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0063 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0064 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0065 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0066 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0067 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0068 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0069 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0070 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0071 : 9.2 : 35 512 512 : 7.57.5e-5 : 3.83.8e-2 : 35 S0072 : 8.1 : 31 512 512 : 7.57.5e-5 : 3.83.8e-2 : 31
---- Ptycho init level 4 ------------------------------------------------------- object size before (1, 4819, 4825) object size before (1, 4819, 4825) object size before (1, 4819, 4825) object size before (1, 4819, 4825) object size after (1, 12181, 15926)
==== Starting DM_pycuda-algorithm. =============================================
Parameter set:
object size after (1, 12181, 15926) object size after (1, 12181, 15926) object size after (1, 12181, 15926)
Thanks for the info. The memory needed to run DM on the GPU is not just the pods, there are some other auxilliary arrays needed that scale with the nr. of modes and the frames per block. It looks like you are using 12 modes in this example and from what is shown above it looks like 35 frames per block were used.
Unfortunately, our internal memory calculation are not that great at the moment that's why you might just this ugly pycuda traceback whenever you run out of memory on the GPU. If it keeps happening, the only thing I can recommend is lowering the frames per block, lowering the modes or increasing the GPU memory, i.e. by scaling the task over multiple GPUs/nodes.
The original issue (megapixel limit) has been fixed with release 0.8. If there are still remaining memory issue for this use case, feel free to open a new issue.
In our reconstruction, we want a large object under 10nm pixel size. However, the object size will be larger than 50Mb. In this case, it will raise the error that arrays larger than 50M are unsupported. Is there any way to fix it? @daurer @bjoernenders
`
raise RuntimeError('Arrays larger than %dM not supported. You ' RuntimeError: Arrays larger than 50M not supported. You requested 193.99M pixels. raise RuntimeError('Arrays larger than %dM not supported. You ' RuntimeError: Arrays larger than 50M not supported. You requested 193.99M pixels. raise RuntimeError('Arrays larger than %dM not supported. You ' RuntimeError: Arrays larger than 50M not supported. You requested 193.99M pixels. raise RuntimeError('Arrays larger than %dM not supported. You ' RuntimeError: Arrays larger than 50M not supported. You requested 193.99M pixels.
Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted.
mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was:
`