Open lcxsnow opened 1 year ago
My cuda and Cudnn verison are 11.4 and 8.2. flax 0.3.6 jax 0.2.27 jaxlib 0.1.76+cuda11.cudnn82
The error message are below.
I0222 12:38:33.308843 140599999719232 xmc_gan.py:119] train_step(batch={'embedding': Traced<ShapedArray(bfloat16[14,17,768])>with<DynamicJaxprTrace(level=0/1)>, 'image': Traced<ShapedArray(bfloat16[14,256,256,3])>with<DynamicJaxprTrace(level=0/1)>, 'image_aug': Traced<ShapedArray(bfloat16[14,256,256,3])>with<DynamicJaxprTrace(level=0/1)>, 'max_len': Traced<ShapedArray(bfloat16[14,1])>with<DynamicJaxprTrace(level=0/1)>, 'sentence_embedding': Traced<ShapedArray(bfloat16[14,768])>with<DynamicJaxprTrace(level=0/1)>, 'z': Traced<ShapedArray(bfloat16[14,128])>with<DynamicJaxprTrace(level=0/1)>}) 2023-02-22 12:39:19.873214: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 64 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.877356: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 64 layout: OutputInputYX shape: 3 3 } {zero_padding: 1 1 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.884515: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 64 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.891635: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.899718: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 128 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.907872: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.912539: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 128 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.917105: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 128 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.920219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 128 input_feature_map_count: 128 layout: OutputInputYX shape: 3 3 } {zero_padding: 1 1 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.925633: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.930975: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 1024 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.934477: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 1024 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.937853: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 1024 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 1024 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.940752: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 256 layout: OutputInputYX shape: 3 3 } {zero_padding: 1 1 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.944959: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 1024 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 1024 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.949473: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 1024 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 2048 input_feature_map_count: 1024 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.952659: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 7 7 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 2048 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.955657: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 2048 spatial: 7 7 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 2048 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.958881: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 7 7 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 512 layout: OutputInputYX shape: 3 3 } {zero_padding: 1 1 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.964249: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 7 7 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 2048 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.969219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 1024 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.973978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 1024 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.981359: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.988299: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 128 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.000546: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 128 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.012110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 64 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.017571: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 64 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.221487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 3 spatial: 229 229 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 3 layout: OutputInputYX shape: 7 7 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.226297: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 57 57 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 128 input_feature_map_count: 128 layout: OutputInputYX shape: 3 3 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.229980: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 29 29 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 256 layout: OutputInputYX shape: 3 3 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.233720: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 15 15 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 512 layout: OutputInputYX shape: 3 3 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. I0222 12:39:35.479767 140599999719232 train_utils.py:436] Finished training step 1. I0222 12:39:37.334168 140599999719232 train_utils.py:436] Finished training step 2. I0222 12:39:38.369166 140599999719232 train_utils.py:436] Finished training step 3. I0222 12:39:39.788250 140599999719232 train_utils.py:436] Finished training step 4. I0222 12:39:41.206766 140599999719232 train_utils.py:436] Finished training step 5. ....
I had tried cudnn 8.6, and it didn't work too.
My cuda and Cudnn verison are 11.4 and 8.2. flax 0.3.6 jax 0.2.27 jaxlib 0.1.76+cuda11.cudnn82
The error message are below.
I0222 12:38:33.308843 140599999719232 xmc_gan.py:119] train_step(batch={'embedding': Traced<ShapedArray(bfloat16[14,17,768])>with<DynamicJaxprTrace(level=0/1)>, 'image': Traced<ShapedArray(bfloat16[14,256,256,3])>with<DynamicJaxprTrace(level=0/1)>, 'image_aug': Traced<ShapedArray(bfloat16[14,256,256,3])>with<DynamicJaxprTrace(level=0/1)>, 'max_len': Traced<ShapedArray(bfloat16[14,1])>with<DynamicJaxprTrace(level=0/1)>, 'sentence_embedding': Traced<ShapedArray(bfloat16[14,768])>with<DynamicJaxprTrace(level=0/1)>, 'z': Traced<ShapedArray(bfloat16[14,128])>with<DynamicJaxprTrace(level=0/1)>}) 2023-02-22 12:39:19.873214: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 64 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.877356: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 64 layout: OutputInputYX shape: 3 3 } {zero_padding: 1 1 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.884515: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 64 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.891635: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.899718: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 128 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.907872: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.912539: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 128 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.917105: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 128 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.920219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 128 input_feature_map_count: 128 layout: OutputInputYX shape: 3 3 } {zero_padding: 1 1 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.925633: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.930975: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 1024 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.934477: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 1024 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.937853: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 1024 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 1024 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.940752: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 256 layout: OutputInputYX shape: 3 3 } {zero_padding: 1 1 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.944959: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 1024 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 1024 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.949473: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 1024 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 2048 input_feature_map_count: 1024 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.952659: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 7 7 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 2048 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.955657: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 2048 spatial: 7 7 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 2048 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.958881: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 7 7 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 512 layout: OutputInputYX shape: 3 3 } {zero_padding: 1 1 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.964249: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 7 7 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 2048 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.969219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 1024 input_feature_map_count: 512 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.973978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 14 14 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 1024 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.981359: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 256 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:19.988299: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 28 28 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 128 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.000546: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 128 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.012110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 64 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.017571: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 64 spatial: 56 56 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 64 layout: OutputInputYX shape: 1 1 } {zero_padding: 0 0 pad_alignment: default filter_strides: 1 1 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.221487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 3 spatial: 229 229 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 64 input_feature_map_count: 3 layout: OutputInputYX shape: 7 7 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.226297: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 128 spatial: 57 57 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 128 input_feature_map_count: 128 layout: OutputInputYX shape: 3 3 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.229980: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 256 spatial: 29 29 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 256 input_feature_map_count: 256 layout: OutputInputYX shape: 3 3 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. 2023-02-22 12:39:20.233720: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:5205] Disabling cuDNN frontend for the following convolution: input: {count: 14 feature_map_count: 512 spatial: 15 15 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} filter: {output_feature_map_count: 512 input_feature_map_count: 512 layout: OutputInputYX shape: 3 3 } {zero_padding: 0 0 pad_alignment: default filter_strides: 2 2 dilation_rates: 1 1 } ... because it uses an identity activation. I0222 12:39:35.479767 140599999719232 train_utils.py:436] Finished training step 1. I0222 12:39:37.334168 140599999719232 train_utils.py:436] Finished training step 2. I0222 12:39:38.369166 140599999719232 train_utils.py:436] Finished training step 3. I0222 12:39:39.788250 140599999719232 train_utils.py:436] Finished training step 4. I0222 12:39:41.206766 140599999719232 train_utils.py:436] Finished training step 5. ....
I had tried cudnn 8.6, and it didn't work too.