search

kdexd / virtex

[CVPR 2021] VirTex: Learning Visual Representations from Textual Annotations
http://kdexd.xyz/virtex
MIT License
556 stars 61 forks source link

unable to find a valid cuDNN algorithm to run convolution #15

Closed Charlie-zhang1406 closed 3 years ago

Charlie-zhang1406 commented 3 years ago

sorry to bother you, but I run into this problem and can not to find a way to fix it. it happens when I train the base virtex model. I have update the cuDNN version into 8.0.3, the former version is 7.6.5. both version have this error.

Charlie-zhang1406 commented 3 years ago

I find the problem exits when apex is doing his thing, when I disable apex and make it compute in fp32, the problem is solved. I still do not know the reason. maybe apex have some version request about cuDNN. I will probably follow this issue uintil I find out the reason.

kdexd commented 3 years ago

Thanks for trying out the code! I am glad that you at least got it working without automatic mixed-precision, which should be perfectly fine if you have enough GPU memory (it will not significantly affect the results or compatibility).

I could not reproduce this issue unfortunately, I am using CUDA 10.1 and CuDNN 8.0.3 As a sanity check, make sure you got NVIDIA Apex installed properly (with CUDA extensions) as mentioned in its README.

Charlie-zhang1406 commented 3 years ago

thank you for your reply, i have reinstalled Apex and it worked.

kdexd commented 3 years ago

Awesome, glad it worked for you!