search

astra-toolbox / astra-toolbox

ASTRA Tomography Toolbox
GNU General Public License v3.0
455 stars 208 forks source link

Crash when using large data sets #24

Closed adler-j closed 8 years ago

adler-j commented 8 years ago

Currently if you give data sets sufficiently large, astra crashes with messages such as

Error: CUDA error 11: invalid argument.
Error: Failed to allocate 300x5000x20 GPU array
Error: CUDA error 11: invalid argument.
Assertion failed: err == cudaSuccess, file c:/Users/wpalenst/git/astra-toolbox/cuda/3d/util3d.cu, line 382

is there any workaround to this available?

Edit: On my machine, with a 980 TI card, the limit seems to be 4096 in any direction.

wjp commented 8 years ago

This is the cuda 3D texture size limit, which is indeed 4096 on compute model 3 (and up) cards.

There's unfortunately no workaround yet, but this will be fixed when I add the capability to (automatically) split volumes in all 3 directions. Currently we only split in the slowest-increasing coordinate index. (So the split blocks are each contiguous.)

adler-j commented 8 years ago

Ok great that you got it in the works, any ETA on this?

wjp commented 8 years ago

After having another look at the code, it seems like I have already done all of the necessary infrastructure, so the basic capability of at least splitting up to below the max texture size should be doable pretty quickly. I'll give it a try next week.

wjp commented 8 years ago

Small status update: this is done, and I plan to merge it tomorrow.

adler-j commented 8 years ago

Very nice, thank you!

wjp commented 8 years ago

This should now be working. Please let us know if you run into further issues.

adler-j commented 8 years ago

Added a small comment to the pull, will try to test this asap.

moosmann commented 6 years ago

Is this issue resolved for the mpi branch or python only? Because I still have the same issue using astra version 1.8.3 and Matlab. Running e.g. the example s020_3d_multiGPU with 4097 angles, astra crashes:

Error: CUDA error 11: invalid argument.
Error: Failed to allocate 1024x4097x171 GPU array
Error: CUDA error 11: invalid argument.
MATLAB: ./../../cuda/3d/util3d.cu:381: bool astraCUDA3d::transferProjectionsToArray(cudaPitchedPtr, 
cudaArray*, const astraCUDA3d::SDimensions3D&): Assertion err == cudaSuccess failed.