weights are old and cannot be used with tensorflow

[mirfan899]

I've tried to run the example but it seems your weights are too old and cannot be used with the TensorFlow version(1.13.1), Keras 2.3.1.

Cuda 10.0.130 Cudnn 7.3.1

Using TensorFlow backend.
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2020-02-27 16:42:29.428823: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-27 16:42:29.435453: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-27 16:42:29.436706: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2bc55c0 executing computations on platform CUDA. Devices:
2020-02-27 16:42:29.436720: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): GeForce RTX 2060 SUPER, Compute Capability 7.5
2020-02-27 16:42:29.471700: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3696000000 Hz
2020-02-27 16:42:29.472057: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x3289690 executing computations on platform Host. Devices:
2020-02-27 16:42:29.472070: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2020-02-27 16:42:29.472140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: GeForce RTX 2060 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:01:00.0
totalMemory: 7.79GiB freeMemory: 7.12GiB
2020-02-27 16:42:29.472150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-02-27 16:42:29.472992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-27 16:42:29.473004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2020-02-27 16:42:29.473008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2020-02-27 16:42:29.473054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6927 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
Found 4 images belonging to 2 classes.
2020-02-27 16:42:31.830396: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2020-02-27 16:42:33.494656: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-02-27 16:42:33.500889: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
  File "/home/irfan/PycharmProjects/MesoNet/example.py", line 25, in <module>
    print('Predicted :', classifier.predict(X), '\nReal class :', y)
  File "/home/irfan/PycharmProjects/MesoNet/classifiers.py", line 14, in predict
    return self.model.predict(x)
  File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/keras/engine/training.py", line 1462, in predict
    callbacks=callbacks)
  File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 324, in predict_loop
    batch_outs = f(ins_batch)
  File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3076, in __call__
    run_metadata=self.run_metadata)
  File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in __call__
    run_metadata_ptr)
  File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[{{node conv2d_1/convolution}}]]

Process finished with exit code 1

[DariusAf]

Hi,

I have just instantiated a model and loaded the pre-trained weights on Tensorflow 2.1 (tf-keras 2.2.4), the prediction works fine, I don't think it is a question of old weights (you could load and re-export them via tf-2.1 if you want to try).

Feels more like a cuda-related problem. Does the model works properly on CPU?

[mirfan899]

I've updated the Cudnn to 7.6.5 and Cuda 10 and TensorFlow 1.15 and it did not work. It seems I have to installed TensorFlow 2.1

[DariusAf]

Any update on your problem? If not I am closing the issue because it seems related to your configuration.

[mirfan899]

Yes, it resolved.