Cuda memory error when using Cuda forward projector

[ObaidullahRahman]

My image size is 1264x1356x1356=2324162304 Projection size: (slice/row, view, column)=(1456,145,1840) My GPU has 80 GB memory. When I use CudaForwardProjectionImageFilter I get memory error (please refer to the attachment).

Simon confirmed that in line 115 of rtkCudaUtilities.cu, my image exceeds the predefined max size of the image. Please increase the predefined max size for images. (I'd like to be able to forward project 2000x2000x2000 image or as much as possible). Thanks.

Best, Obaid

[SimonRit]

Dear Obaid, Can you test the fix https://github.com/RTKConsortium/RTK/pull/598 that I have implemented? The new Python wheel can be downloaded from the CI if you're logged in Github, at the bottom of the workflow page https://github.com/RTKConsortium/RTK/actions/runs/9051385855. Simon

On Mon, Apr 29, 2024 at 7:47 PM ObaidullahRahman @.***> wrote:

My image size is 1264x1356x1356=2324162304 Projection size: (slice/row, view, column)=(1456,145,1840) My GPU has 80 GB memory. When I use CudaForwardProjectionImageFilter I get memory error (please refer to the attachment).

Simon confirmed that in line 115 of rtkCudaUtilities.cu, my image exceeds the predefined max size of the image. Please increase the predefined max size for images. (I'd like to be able to forward project 2000x2000x2000 image or as much as possible). Thanks.

Best, Obaid Screenshot.2024-04-24.at.8.34.20.PM.png (view on web) https://github.com/RTKConsortium/RTK/assets/36867277/e6554b6c-d682-4f0c-b41b-826685434683

— Reply to this email directly, view it on GitHub https://github.com/RTKConsortium/RTK/issues/594, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACP64FO7Z4WKAYA2BSKCDDY72BTFAVCNFSM6AAAAABG6YWQZKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DSNRQHAYTCNI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ObaidullahRahman]

Hi Simon, I was able to install it. I also had to install the right itk (5.3.0) but that is beside the point. I am getting the exact same error as before. -Obaid

[SimonRit]

That does not seem to be the right installation. In a fresh environment, the requirement is ITK 5.4rc1, which can be installed with pip install itk==5.4rc1. Then make sure to install the package from the CI which does not seem to be what you have installed (rtkCudaUtilities.cu line 115 does not point to an exception).

[ObaidullahRahman]

When I install from the CI, this is error I get.

[ObaidullahRahman]

When I install from the CI, this is error I get.

That's why I have to downgrade to itk 5.3.0

[SimonRit]

If you downgrade, you cannot test the fix! Segmentation faults occur for me when I don't have a matching ITK version or if I do not install the correct linux version. There are two linux versions, 2014 for older versions and 2_28 for newer ones, have you tested both? BTW, ITK has released v5.4.0, you can test the new CudaCommon and RTK packages https://github.com/RTKConsortium/ITKCudaCommon/actions/runs/9169210647 https://github.com/RTKConsortium/RTK/actions/runs/9169284466 Python 3.11 packages are not compiled yet but you can test one of the other Python versions.

[SimonRit]

The linux version depends on your OS version, see https://github.com/pypa/manylinux.

[ObaidullahRahman]

I was able to install this one:

The itk that gets installed in indeed 5.4.0. But now I get a different error:

[SimonRit]

I'm surprised that you did not have that before. We do not ship the cuda libraries with the package, you have to install cuda 11.6 on your computer. You can have multiple versions of cuda on the same computer. For windows, you can add the cuda path like this: https://github.com/RTKConsortium/RTK/blob/master/examples/FirstReconstruction/FirstCudaReconstruction.py#L13-L14. For linux, add the cuda lib directory (containing libcudart.so.11.0 among others) to LD_LIBRARY_PATH. Note that you can also opt for Cuda 12.1 now.

[ObaidullahRahman]

I think I have cuda installed. This is what nvidia-smi returns:

[SimonRit]

This just indicates the Cuda compability of your driver. Which OS are you using? The package you have installed requires Cuda 11.6, ot Cuda 12.2. You may want to try the Cuda 12.1 version, I have experienced incompatibilities between Cuda 11.6 and a Cuda 12 driver.