cudnn check failed

[chenchr]

Hello mayern. I not sure if it's ok to report issue here. I download the dispflownet-release project from uni-freiburg and try to train flownets from stratch. However, I found that when using cudnn6 or cudnn7, the code can't compile. I am using cuda 8.0, them I try cudnn5, the code can compile, however, when launch caffe.bin to train, a error occurs:

F1203 03:40:45.984501 12374 cudnn.hpp:113] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0)  CUDNN_STATUS_BAD_PARAM

I check the code in cudnn.hpp:113,which is the same as the code in current flownet2:

CUDNN_CHECK(cudnnSetConvolution2dDescriptor(*conv,
      pad_h, pad_w, stride_h, stride_w, 1, 1, CUDNN_CROSS_CORRELATION));

Besides, when I comment out USE_CUDNN to not to use it, the compiling still report error. As I think you are also the author of dispnet, and these three projects may use same code base, and the dispflownet-release project is for download only, it's not on github, therefore I have to submit a issue here. Update here. I use the code in the repository and still get the same error.. Thanks.

[nikolausmayer]

Hi, it's fine to post this issue here, but the problem might not affect the flownet2 branch anymore.

Is it possible that the code compiles with CuDNN5, but at runtime a libcudnn.so from another version is loaded?

[chenchr]

hmm, I check the file, when I install cudnn5, I have removed the file corresponding to cudnn6 and 7. Did you use cuda 8.0?

[nikolausmayer]

Yes, we originally used cuda 7.5, but e.g. the codebase of our Docker image compiles with cuda 8.0.

[chenchr]

OK. Maybe I should have a try to use docker.

[nikolausmayer]

It's the same codebase IIRC, so there should not be a difference. I just don't remember compiling it with cuda 8 outside of Docker.

[chenchr]

Forgot to feedback... I've solve the problem. As you said, the code compiles with right cudnn lib but links to wrong one.. As there are cudnn7.0 in /usr/lib... I remove that file and it works. Thanks!