Help: Failed to get convolution algorithm

2021-09-22 08:06:14.234343: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
fps : 60
2021-09-22 08:06:16.272909: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-09-22 08:06:16.294530: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 08:06:16.295305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2021-09-22 08:06:16.295393: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-09-22 08:06:16.299668: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-09-22 08:06:16.299764: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-09-22 08:06:16.301051: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-09-22 08:06:16.301799: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-09-22 08:06:16.305778: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-09-22 08:06:16.306866: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-09-22 08:06:16.307181: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-09-22 08:06:16.307336: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 08:06:16.308216: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 08:06:16.309021: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-09-22 08:06:16.309402: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-09-22 08:06:16.309722: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 08:06:16.310485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2021-09-22 08:06:16.310635: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 08:06:16.311429: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 08:06:16.312205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-09-22 08:06:16.312300: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-09-22 08:06:16.884681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-09-22 08:06:16.884740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2021-09-22 08:06:16.884767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2021-09-22 08:06:16.885046: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 08:06:16.886096: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 08:06:16.887106: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 08:06:16.887864: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-09-22 08:06:16.887939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10800 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
layer24 output shape: 256 360 640
Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 3, 360, 640)]     0         
_________________________________________________________________
conv2d (Conv2D)              (None, 64, 360, 640)      1792      
_________________________________________________________________
activation (Activation)      (None, 64, 360, 640)      0         
_________________________________________________________________
batch_normalization (BatchNo (None, 64, 360, 640)      2560      
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 64, 360, 640)      36928     
_________________________________________________________________
activation_1 (Activation)    (None, 64, 360, 640)      0         
_________________________________________________________________
batch_normalization_1 (Batch (None, 64, 360, 640)      2560      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 64, 180, 320)      0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 128, 180, 320)     73856     
_________________________________________________________________
activation_2 (Activation)    (None, 128, 180, 320)     0         
_________________________________________________________________
batch_normalization_2 (Batch (None, 128, 180, 320)     1280      
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 128, 180, 320)     147584    
_________________________________________________________________
activation_3 (Activation)    (None, 128, 180, 320)     0         
_________________________________________________________________
batch_normalization_3 (Batch (None, 128, 180, 320)     1280      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 128, 90, 160)      0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 256, 90, 160)      295168    
_________________________________________________________________
activation_4 (Activation)    (None, 256, 90, 160)      0         
_________________________________________________________________
batch_normalization_4 (Batch (None, 256, 90, 160)      640       
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 256, 90, 160)      590080    
_________________________________________________________________
activation_5 (Activation)    (None, 256, 90, 160)      0         
_________________________________________________________________
batch_normalization_5 (Batch (None, 256, 90, 160)      640       
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 256, 90, 160)      590080    
_________________________________________________________________
activation_6 (Activation)    (None, 256, 90, 160)      0         
_________________________________________________________________
batch_normalization_6 (Batch (None, 256, 90, 160)      640       
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 256, 45, 80)       0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 512, 45, 80)       1180160   
_________________________________________________________________
activation_7 (Activation)    (None, 512, 45, 80)       0         
_________________________________________________________________
batch_normalization_7 (Batch (None, 512, 45, 80)       320       
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 512, 45, 80)       2359808   
_________________________________________________________________
activation_8 (Activation)    (None, 512, 45, 80)       0         
_________________________________________________________________
batch_normalization_8 (Batch (None, 512, 45, 80)       320       
_________________________________________________________________
conv2d_9 (Conv2D)            (None, 512, 45, 80)       2359808   
_________________________________________________________________
activation_9 (Activation)    (None, 512, 45, 80)       0         
_________________________________________________________________
batch_normalization_9 (Batch (None, 512, 45, 80)       320       
_________________________________________________________________
up_sampling2d (UpSampling2D) (None, 512, 90, 160)      0         
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 256, 90, 160)      1179904   
_________________________________________________________________
activation_10 (Activation)   (None, 256, 90, 160)      0         
_________________________________________________________________
batch_normalization_10 (Batc (None, 256, 90, 160)      640       
_________________________________________________________________
conv2d_11 (Conv2D)           (None, 256, 90, 160)      590080    
_________________________________________________________________
activation_11 (Activation)   (None, 256, 90, 160)      0         
_________________________________________________________________
batch_normalization_11 (Batc (None, 256, 90, 160)      640       
_________________________________________________________________
conv2d_12 (Conv2D)           (None, 256, 90, 160)      590080    
_________________________________________________________________
activation_12 (Activation)   (None, 256, 90, 160)      0         
_________________________________________________________________
batch_normalization_12 (Batc (None, 256, 90, 160)      640       
_________________________________________________________________
up_sampling2d_1 (UpSampling2 (None, 256, 180, 320)     0         
_________________________________________________________________
conv2d_13 (Conv2D)           (None, 128, 180, 320)     295040    
_________________________________________________________________
activation_13 (Activation)   (None, 128, 180, 320)     0         
_________________________________________________________________
batch_normalization_13 (Batc (None, 128, 180, 320)     1280      
_________________________________________________________________
conv2d_14 (Conv2D)           (None, 128, 180, 320)     147584    
_________________________________________________________________
activation_14 (Activation)   (None, 128, 180, 320)     0         
_________________________________________________________________
batch_normalization_14 (Batc (None, 128, 180, 320)     1280      
_________________________________________________________________
up_sampling2d_2 (UpSampling2 (None, 128, 360, 640)     0         
_________________________________________________________________
conv2d_15 (Conv2D)           (None, 64, 360, 640)      73792     
_________________________________________________________________
activation_15 (Activation)   (None, 64, 360, 640)      0         
_________________________________________________________________
batch_normalization_15 (Batc (None, 64, 360, 640)      2560      
_________________________________________________________________
conv2d_16 (Conv2D)           (None, 64, 360, 640)      36928     
_________________________________________________________________
activation_16 (Activation)   (None, 64, 360, 640)      0         
_________________________________________________________________
batch_normalization_16 (Batc (None, 64, 360, 640)      2560      
_________________________________________________________________
conv2d_17 (Conv2D)           (None, 256, 360, 640)     147712    
_________________________________________________________________
activation_17 (Activation)   (None, 256, 360, 640)     0         
_________________________________________________________________
batch_normalization_17 (Batc (None, 256, 360, 640)     2560      
_________________________________________________________________
reshape (Reshape)            (None, 256, 230400)       0         
_________________________________________________________________
permute (Permute)            (None, 230400, 256)       0         
_________________________________________________________________
activation_18 (Activation)   (None, 230400, 256)       0         
=================================================================
Total params: 10,719,104
Trainable params: 10,707,744
Non-trainable params: 11,360
_________________________________________________________________
2021-09-22 08:06:17.330092: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open WeightsTracknet/model.1: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
OpenCV: FFMPEG: tag 0x44495658/'XVID' is not supported with codec id 12 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x7634706d/'mp4v'
Using device cuda
Detecting the court and the players...
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
BOXES  [array([452.1675 , 735.3489 , 572.7479 , 954.95905], dtype=float32)]
BIGGEST  [452. 735. 573. 955.]
Finished!
Tracking the ball: 0.0
2021-09-22 08:20:51.717046: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-09-22 08:20:51.724503: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2299995000 Hz
2021-09-22 08:20:52.243946: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-09-22 08:20:54.014311: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2021-09-22 08:20:54.015598: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2021-09-22 08:20:54.015950: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_ops_fused_impl.h:698 : Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
Traceback (most recent call last):
  File "predict_video.py", line 155, in <module>
    pr = m.predict(np.array([X]))[0]
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1727, in predict
    tmp_batch_outputs = self.predict_function(iterator)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 957, in _call
    filtered_flat_args, self._concrete_stateful_fn.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1961, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[node model_1/activation/Relu (defined at predict_video.py:155) ]] [Op:__inference_predict_function_1776]

Function call stack:
predict_function
😳😳😳😳
ArtLabss / tennis-tracking

Help: Failed to get convolution algorithm #6
