search

MRtrix3 / mrtrix3

MRtrix3 provides a set of tools to perform various advanced diffusion MRI analyses, including constrained spherical deconvolution (CSD), probabilistic tractography, track-density imaging, and apparent fibre density
http://www.mrtrix.org
Mozilla Public License 2.0
293 stars 180 forks source link

CUDA not working in Singularity #2383

Open araikes opened 3 years ago

araikes commented 3 years ago

Describe the bug

I downloaded the official Singularity image to use with some data. Given that CUDA paths were defined in the Singularity recipe, I anticipated dwifslpreproc to make use of eddy_cuda. However, this is not the case because there are no CUDA folders in the actual Singularity container and eddy_cuda9.1 requires CUDA 9.1 drivers, which are not available in Ubuntu 20.04 (the base OS for the container).

If you are running this on a system with an older OS that has CUDA 9.1 toolkit, it may work by default when you load the container with the --nv flag, but if you're running anything newer (e.g., our HPC is running CUDA 11.4 with no backports) then this does not work out of the box.

I tested locally and including the following (created by others, not me) in the recipe enables CUDA support on our HPC. https://gist.github.com/DaneGardner/accd6fd330348543167719002a661bd5#install-cuda-toolkit

To Reproduce

Try running eddy_cuda on a GPU running anything later than CUDA 9.1.

Platform/Environment/Version

Please provide the following information:

Lestropie commented 3 years ago

Cheers Adam. Once I had succeeded in getting CUDA working myself on my local HPC based on sysadmin advice (which included explicitly loading a cuda module of the required version), I didn't invest any further effort. If it might be possible to make the CUDA support a bit more robust, then it's worth experimenting with. But I'll need to test a proposed change across a few different systems.

Did you want to generate yourself the first commit with the basics of the proposed change (see contribution documentation)? That way you'd get credit for it, even if I have to tweak things subsequently.

araikes commented 3 years ago

@Lestropie,

I will work on generating the change. Happy to do it.

araikes commented 3 years ago

PR created, but I just realized I commented out some of the rm statement at the end so I could build it using Sylabs remote. That can be fixed depending on how/where you need to build for testing.