microsoft / CNTK

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit
https://docs.microsoft.com/cognitive-toolkit/
Other
17.49k stars 4.3k forks source link

A model causes CNTK crash with cudnnSetPoolingNDDescriptor when invoking the pooling operator #3869

Open rubbberrabbit opened 1 year ago

rubbberrabbit commented 1 year ago

Hi, we find a VGG model that can cause CNTK crash with cudnnSetPoolingNDDescriptor when invoking the pooling operator in the CNTK below 2.7.0 version. The model may have generated an incorrect parameter during pooling and lacked reasonable error hints. We provide the model file and replay script at this link https://drive.google.com/drive/folders/1RJ7vCO04I2HXgKdHAK0kOaY4OdDJ4Mt7?usp=sharing . We find this bug in CNTK-GPU version 2.7.0 with the keras version 2.2.4. The program execution stack in the crash is as follows:

image