Closed PerloDaniele closed 3 years ago
Thanks, we will check it
Hello, Would you mind to check at the model initialization if there is any warning about padding that is not supported by cuDNN?
Thank you
Adrián Castelló
El 23 abr 2021, a las 18:52, Roberto Paredes Palacios @.***> escribió:
Thanks, we will check it
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.
Yep, there are some warnings
Building model
CS with low memory setup
Selecting GPU device 0
EDDLL is running on GPU device 0, GeForce RTX 2080 Ti
CuBlas initialized on GPU device 0, GeForce RTX 2080 Ti
CuRand initialized on GPU device 0, GeForce RTX 2080 Ti
CuDNN initialized on GPU device 0, GeForce RTX 2080 Ti
Warning: asymmetric padding not supported by cuDNN... fixing ... potential shapes mismatch later
Warning: asymmetric padding not supported by cuDNN... fixing ... potential shapes mismatch later
Warning: asymmetric padding not supported by cuDNN... fixing ... potential shapes mismatch later
Warning: asymmetric padding not supported by cuDNN... fixing ... potential shapes mismatch later
-------------------------------------------------------------------------------
Ok. That is because asymmetric padding is nor supported by cuDNN. Sometimes, EDDL generates padding for top but not for the bottom (or left but not right-side) of the data and, EDDL using cuDNN fixes it by adding padding. This feature will be fixed in the next release and the execution will abort if asymmetric padding is detected.
Right now, you should check the padding or to add it manually.
Adrián Castelló
El 24 abr 2021, a las 11:50, PerloDaniele @.***> escribió:
Yep, there are some warnings
Building model CS with low memory setup Selecting GPU device 0 EDDLL is running on GPU device 0, GeForce RTX 2080 Ti CuBlas initialized on GPU device 0, GeForce RTX 2080 Ti CuRand initialized on GPU device 0, GeForce RTX 2080 Ti CuDNN initialized on GPU device 0, GeForce RTX 2080 Ti Warning: asymmetric padding not supported by cuDNN... fixing ... potential shapes mismatch later Warning: asymmetric padding not supported by cuDNN... fixing ... potential shapes mismatch later Warning: asymmetric padding not supported by cuDNN... fixing ... potential shapes mismatch later Warning: asymmetric padding not supported by cuDNN... fixing ... potential shapes mismatch later
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.
Fixed.
CUDNN error by running UNet on docker dhealth/pylibs-toolkit:0.10.0-cudnn
Model:
By using dhealth/pylibs-toolkit:0.10.0-gpu the model training works fine on gpu.