RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling `cusolverDnCreate(handle)

Alexandrsv commented 1 year ago

python generate.py -s 512 512 -p "A painting of an apple in a fruit bowl"

It worked correctly before. Other neural networks work correctly. Maybe the reason is the video card update 1080 -> 4090 How to fix the error?

/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/pytorch_lightning/metrics/__init__.py:43: LightningDeprecationWarning: `pytorch_lightning.metrics.*` module has been renamed to `torchmetrics.*` and split off to its own package (https://github.com/PyTorchLightning/metrics) since v1.3 and will be removed in v1.5
  rank_zero_deprecation(
/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/augmentation/augmentation.py:1830: DeprecationWarning: GaussianBlur is no longer maintained and will be removed from the future versions. Please use RandomGaussianBlur instead.
  warnings.warn(
Working with z of shape (1, 256, 16, 16) = 65536 dimensions.
loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
VQLPIPSWithDiscriminator running with hinge loss.
Restored from checkpoints/vqgan_imagenet_f16_16384.ckpt
Using device: cuda:0
Optimising using: Adam
Using text prompts: ['A painting of an apple in a fruit bowl']
Using seed: 162421434451382733
0it [00:00, ?it/s]
Traceback (most recent call last):
  File "/run/media/alexandr/12T/Pytochnaya/VQGAN-CLIP/generate.py", line 862, in <module>
    train(i)
  File "/run/media/alexandr/12T/Pytochnaya/VQGAN-CLIP/generate.py", line 743, in train
    lossAll = ascend_txt()
  File "/run/media/alexandr/12T/Pytochnaya/VQGAN-CLIP/generate.py", line 722, in ascend_txt
    iii = perceptor.encode_image(normalize(make_cutouts(out))).float()
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/run/media/alexandr/12T/Pytochnaya/VQGAN-CLIP/generate.py", line 355, in forward
    batch = self.augs(torch.cat(cutouts, dim=0))
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/augmentation/base.py", line 247, in forward
    output = self.apply_func(in_tensor, in_transform, self._params, return_transform)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/augmentation/base.py", line 212, in apply_func
    output[to_apply] = self.apply_transform(in_tensor[to_apply], params, trans_matrix[to_apply])
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/augmentation/augmentation.py", line 685, in apply_transform
    return warp_affine(
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/geometry/transform/imgwarp.py", line 192, in warp_affine
    dst_norm_trans_src_norm: torch.Tensor = normalize_homography(M_3x3, (H, W), dsize)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/geometry/transform/homography_warper.py", line 396, in normalize_homography
    src_pix_trans_src_norm = _torch_inverse_cast(src_norm_trans_src_pix)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/utils/helpers.py", line 48, in _torch_inverse_cast
    return torch.inverse(input.to(dtype)).to(input.dtype)
RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling `cusolverDnCreate(handle)

ZJChe commented 11 months ago

Hi, I just ran into the same issue with 4090 video card. I finally managed to make it work with following modifications to the setup in README

pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 -f https://download.pytorch.org/whl/torch_stable.html

And:

pip install kornia==0.5.10

Don't know whether you still need it, just take a note in case someone else come across the same problem 😄

lyl7650 commented 6 months ago

Hi, I just ran into the same issue with 4090 video card. I finally managed to make it work with following modifications to the setup in README
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 -f https://download.pytorch.org/whl/torch_stable.html
And:
pip install kornia==0.5.10
Don't know whether you still need it, just take a note in case someone else come across the same problem 😄

it works to me

nerdyrodent / VQGAN-CLIP

RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling `cusolverDnCreate(handle) #164