Closed ltang320 closed 1 year ago
@ltang320 Thanks for sharing the details and sorry that we could not have a look at this during the developer meeting. Could you try with a smaller frames_per_block
something like 100
or even 10
. This will break up the data and model into smaller chunks and should help avoiding the GPU memory issues that you have reported above.
The DM_pycuda engine is capable of handling large data that does not all fit into the memory of a single GPU by using CUDA streams, but a single chunk (frames_per_block) still needs to fit into GPU memory therefore reducing this parameter should help.
@ltang320 Thanks for sharing the details and sorry that we could not have a look at this during the developer meeting. Could you try with a smaller
frames_per_block
something like100
or even10
. This will break up the data and model into smaller chunks and should help avoiding the GPU memory issues that you have reported above.The DM_pycuda engine is capable of handling large data that does not all fit into the memory of a single GPU by using CUDA streams, but a single chunk (frames_per_block) still needs to fit into GPU memory therefore reducing this parameter should help.
@daurer Thanks for your reply. It works for frames_per_block = 15
. From my current testing, the minimum frame_per_block should be above 10. For each pods, the memory is smaller than 500MB and it could work properly.
Thanks a lot for help from both of you. @bjoernenders @daurer
For a dataset with 47,206 frames, I prepare cropping 256x256 pixels for each diffraction pattern and analyze it with 4 modes. This means it generates 188,824 pods in the memory. Loading the data is no problem and I use p.frames_per_block = 1000 to load it separately.
However, when it starts doing the reconstruction, there is the following problem generated which is related to the memory issue: ` ==== Starting DM_pycuda-algorithm. =============================================
` Could you help me with this problem? Thanks a lot! @daurer @bjoernenders