Skipping ERROR caught in nll = model.optimize_parameters(current_step): svd_cuda: the updating process of SBDSDC did not converge (error: 23)

[flybiubiu]

Thx author!I train x4 is ok! But when I train x8: Skipping ERROR caught in nll = model.optimize_parameters(current_step): svd_cuda: the updating process of SBDSDC did not converge (error: 23)

Python 3.8.5 (default, Sep 4 2020, 07:30:14) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.

import torch print(torch.version) 1.7.1+cu110 print(torch.version.cuda) 11.0

print(torch.backends.cudnn.version()) 8005

············································································································ My GPU is 3090.I run setup code and find the cuda version is not compare.After that I reinstall with (pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2

About iters is 10000.

[RedRAINXXXX]

I encountered the same problem as you. When this error occurs, subsequent data will have this error

[JingzheLyp]

Hi, I encountered the same problem as you. Have you solved the problem? @flybiubiu, @RedRAINXXXX

[RedRAINXXXX]

Hi, I encountered the same problem as you. Have you solved the problem? @flybiubiu, @RedRAINXXXX

Perhaps because the learning rate is too high, you can try warm up or lower the learning rate directly