Training Error - 'an illegal memory access was encountered'

skhu101 / GauHuman

Code for our CVPR'2024 paper "GauHuman: Articulated Gaussian Splatting from Monocular Human Videos"

Other

322 stars 27 forks source link

Training Error - 'an illegal memory access was encountered' #21

Closed zfonemore closed 7 months ago

zfonemore commented 7 months ago

Hi, I run the training code successfully, but after 30 iterations' training it crashes. The error occurs in the loss.backward(), and it raise an error "RuntimeError: merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered". I have checked the losses, and it seems normal. Can you help me? Thanks a lot.

zfonemore commented 7 months ago

The problem is caused by that I use CUDA 11.4. After I upgrade CUDA to 11.8, the problem is solved.