test_fft.py fails with 2023.1.1 on a GTX 1080 Ti

[luc-j-bourhis]

Here is the relevant output of test_fft.py

test_c2c (__main__.TestFFT.test_c2c)
Run C2C tests ... nvrtcCompileProgram error: NVRTC_ERROR_BUILTIN_OPERATION_FAILURE
nvrtc: error: failed to load builtins for compute_30.
extern __shared__ float shared[];
extern "C" __global__ void __launch_bounds__(6) VkFFT_main (float2* inputs, float2* outputs) {

For a GTX 1080 Ti, it should use compute_61 afaiu.

nvidia-smi says

Driver Version: 530.30.02    CUDA Version: 12.1

and nvcc --version gives

Cuda compilation tools, release 10.1, V10.1.243

Any help will be appreciated.

[vincefn]

Hi Luc, sorry for not seeing this earlier - I'm not sure what the exact issue is, but given the discrepancy between the toolkit (nvcc 10.1.243) and driver (12.1), I would suggest to update the toolkit.

I'm testing using a GTX 1080 before release, so I know there should be no issues with that card (see the test in the current dev branch: http://ftp.esrf.fr/pub/scisoft/PyNX/pyvkfft-test/pyvkfft-test-2023-07-28-gtx1080ti_cu/pyvkfft-test.html), though I'm mostly testing with cuda toolkits >=11.7 now.

[luc-j-bourhis]

Hi Vincent. Thanks a bunch ! Installing cuda-nvcc-12-1 solved the problem indeed.

Just in case someone has the same problem, I shall add that it got installed in /usr/local/cuda-12.1, so the older nvcc in /usr/bin needs to be hidden by changing PATH, and LD_LIBRARY_PATH too for good measure.

Thanks again!

[vincefn]

Great ! It should also be possible to rely on the cuda toolkit available trough conda (I think it's automatically installed by the pyvkfft conda-forge package).