RuntimeError: CUDA error: the provided PTX was compiled with an unsupported toolchain.

[sylyt62]

Struggled for days.. looking for helps :(

Everything went well on windows and ubuntu with an RTX gpu, but met a strange error while testing with a Tesla V100:

ubuntu 18.04, python 3.10, tesla V100, cuda 11.7, torch 2.0.1

minimum reproducing:

import modules.radnerfs.raymarching as raymarching
from time import time
import numpy as np
import torch

if __name__ == '__main__':
    t = time()
    data = np.random.random((10, 10, 3))
    print('Numpy to tensor....')
    data_ts = torch.from_numpy(data)
    print('tensor to cuda ...')
    data_ts = data_ts.cuda()
    print(f'Done. {time() - t:.2f}')

Error logs:

Numpy to tensor.... tensor to cuda ... Traceback (most recent call last): File "/data/workspace/online-lipsync/online-genefacepp/debug2.py", line 14, in data_ts = data_ts.cuda() RuntimeError: CUDA error: the provided PTX was compiled with an unsupported toolchain. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[sylyt62]

fixed by update nvidia driver from 510 to 515.