search

theAIGuysCode / Object-Detection-API

Yolov3 Object Detection implemented as APIs, using TensorFlow and Flask
Apache License 2.0
339 stars 179 forks source link

Error Message When Loading My Weights #3

Closed rob26r closed 4 years ago

rob26r commented 4 years ago

I was able to follow the entire video but getting the error messag below when loading this,

(yolov3-gpu) C:\Users\rob26\Desktop\Object-Detection-API>python load_weights.py

(As suggested, using Anaconda)...

(yolov3-gpu) C:\Users\rob26\Desktop\Object-Detection-API>python load_weights.py
2020-03-22 14:12:34.123439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2020-03-22 14:12:36.164552: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-03-22 14:12:36.190347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1660 major: 7 minor: 5 memoryClockRate(GHz): 1.83
pciBusID: 0000:01:00.0
2020-03-22 14:12:36.196564: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-03-22 14:12:36.201758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-22 14:12:36.210883: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-03-22 14:12:36.216014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1660 major: 7 minor: 5 memoryClockRate(GHz): 1.83
pciBusID: 0000:01:00.0
2020-03-22 14:12:36.221990: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-03-22 14:12:36.226487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-22 14:12:36.784693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-22 14:12:36.788336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2020-03-22 14:12:36.791547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
2020-03-22 14:12:36.796796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4630 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1660, pci bus id: 0000:01:00.0, compute capability: 7.5)
Model: "yolov3"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input (InputLayer)              [(None, None, None,  0
__________________________________________________________________________________________________
yolo_darknet (Model)            ((None, None, None,  40620640    input[0][0]
__________________________________________________________________________________________________
yolo_conv_0 (Model)             (None, None, None, 5 11024384    yolo_darknet[1][2]
__________________________________________________________________________________________________
yolo_conv_1 (Model)             (None, None, None, 2 2957312     yolo_conv_0[1][0]
                                                                 yolo_darknet[1][1]
__________________________________________________________________________________________________
yolo_conv_2 (Model)             (None, None, None, 1 741376      yolo_conv_1[1][0]
                                                                 yolo_darknet[1][0]
__________________________________________________________________________________________________
yolo_output_0 (Model)           (None, None, None, 3 4984063     yolo_conv_0[1][0]
__________________________________________________________________________________________________
yolo_output_1 (Model)           (None, None, None, 3 1312511     yolo_conv_1[1][0]
__________________________________________________________________________________________________
yolo_output_2 (Model)           (None, None, None, 3 361471      yolo_conv_2[1][0]
__________________________________________________________________________________________________
yolo_boxes_0 (Lambda)           ((None, None, None,  0           yolo_output_0[1][0]
__________________________________________________________________________________________________
yolo_boxes_1 (Lambda)           ((None, None, None,  0           yolo_output_1[1][0]
__________________________________________________________________________________________________
yolo_boxes_2 (Lambda)           ((None, None, None,  0           yolo_output_2[1][0]
__________________________________________________________________________________________________
yolo_nms (Lambda)               ((None, 100, 4), (No 0           yolo_boxes_0[0][0]
                                                                 yolo_boxes_0[0][1]
                                                                 yolo_boxes_0[0][2]
                                                                 yolo_boxes_1[0][0]
                                                                 yolo_boxes_1[0][1]
                                                                 yolo_boxes_1[0][2]
                                                                 yolo_boxes_2[0][0]
                                                                 yolo_boxes_2[0][1]
                                                                 yolo_boxes_2[0][2]
==================================================================================================
Total params: 62,001,757
Trainable params: 61,949,149
Non-trainable params: 52,608
__________________________________________________________________________________________________
I0322 14:12:41.026230  8916 load_weights.py:19] model created
I0322 14:12:41.028251  8916 utils.py:47] yolo_darknet/conv2d bn
I0322 14:12:41.031241  8916 utils.py:47] yolo_darknet/conv2d_1 bn
I0322 14:12:41.033211  8916 utils.py:47] yolo_darknet/conv2d_2 bn
I0322 14:12:41.036204  8916 utils.py:47] yolo_darknet/conv2d_3 bn
I0322 14:12:41.039179  8916 utils.py:47] yolo_darknet/conv2d_4 bn
I0322 14:12:41.042149  8916 utils.py:47] yolo_darknet/conv2d_5 bn
I0322 14:12:41.045166  8916 utils.py:47] yolo_darknet/conv2d_6 bn
I0322 14:12:41.047483  8916 utils.py:47] yolo_darknet/conv2d_7 bn
I0322 14:12:41.050596  8916 utils.py:47] yolo_darknet/conv2d_8 bn
I0322 14:12:41.052566  8916 utils.py:47] yolo_darknet/conv2d_9 bn
I0322 14:12:41.057577  8916 utils.py:47] yolo_darknet/conv2d_10 bn
I0322 14:12:41.060544  8916 utils.py:47] yolo_darknet/conv2d_11 bn
I0322 14:12:41.065555  8916 utils.py:47] yolo_darknet/conv2d_12 bn
I0322 14:12:41.067526  8916 utils.py:47] yolo_darknet/conv2d_13 bn
I0322 14:12:41.071515  8916 utils.py:47] yolo_darknet/conv2d_14 bn
I0322 14:12:41.074537  8916 utils.py:47] yolo_darknet/conv2d_15 bn
I0322 14:12:41.079493  8916 utils.py:47] yolo_darknet/conv2d_16 bn
I0322 14:12:41.082505  8916 utils.py:47] yolo_darknet/conv2d_17 bn
I0322 14:12:41.086475  8916 utils.py:47] yolo_darknet/conv2d_18 bn
I0322 14:12:41.089491  8916 utils.py:47] yolo_darknet/conv2d_19 bn
I0322 14:12:41.094455  8916 utils.py:47] yolo_darknet/conv2d_20 bn
I0322 14:12:41.097445  8916 utils.py:47] yolo_darknet/conv2d_21 bn
I0322 14:12:41.101435  8916 utils.py:47] yolo_darknet/conv2d_22 bn
I0322 14:12:41.104452  8916 utils.py:47] yolo_darknet/conv2d_23 bn
I0322 14:12:41.109441  8916 utils.py:47] yolo_darknet/conv2d_24 bn
I0322 14:12:41.112406  8916 utils.py:47] yolo_darknet/conv2d_25 bn
I0322 14:12:41.116420  8916 utils.py:47] yolo_darknet/conv2d_26 bn
I0322 14:12:41.129360  8916 utils.py:47] yolo_darknet/conv2d_27 bn
I0322 14:12:41.132377  8916 utils.py:47] yolo_darknet/conv2d_28 bn
I0322 14:12:41.144320  8916 utils.py:47] yolo_darknet/conv2d_29 bn
I0322 14:12:41.148334  8916 utils.py:47] yolo_darknet/conv2d_30 bn
I0322 14:12:41.160302  8916 utils.py:47] yolo_darknet/conv2d_31 bn
I0322 14:12:41.163483  8916 utils.py:47] yolo_darknet/conv2d_32 bn
I0322 14:12:41.175453  8916 utils.py:47] yolo_darknet/conv2d_33 bn
I0322 14:12:41.178956  8916 utils.py:47] yolo_darknet/conv2d_34 bn
I0322 14:12:41.189929  8916 utils.py:47] yolo_darknet/conv2d_35 bn
I0322 14:12:41.194916  8916 utils.py:47] yolo_darknet/conv2d_36 bn
I0322 14:12:41.205785  8916 utils.py:47] yolo_darknet/conv2d_37 bn
I0322 14:12:41.209749  8916 utils.py:47] yolo_darknet/conv2d_38 bn
I0322 14:12:41.220768  8916 utils.py:47] yolo_darknet/conv2d_39 bn
I0322 14:12:41.224793  8916 utils.py:47] yolo_darknet/conv2d_40 bn
I0322 14:12:41.236654  8916 utils.py:47] yolo_darknet/conv2d_41 bn
I0322 14:12:41.239645  8916 utils.py:47] yolo_darknet/conv2d_42 bn
I0322 14:12:41.251588  8916 utils.py:47] yolo_darknet/conv2d_43 bn
I0322 14:12:41.305469  8916 utils.py:47] yolo_darknet/conv2d_44 bn
I0322 14:12:41.312425  8916 utils.py:47] yolo_darknet/conv2d_45 bn
I0322 14:12:41.363314  8916 utils.py:47] yolo_darknet/conv2d_46 bn
I0322 14:12:41.370305  8916 utils.py:47] yolo_darknet/conv2d_47 bn
I0322 14:12:41.421196  8916 utils.py:47] yolo_darknet/conv2d_48 bn
I0322 14:12:41.428506  8916 utils.py:47] yolo_darknet/conv2d_49 bn
I0322 14:12:41.480397  8916 utils.py:47] yolo_darknet/conv2d_50 bn
I0322 14:12:41.488376  8916 utils.py:47] yolo_darknet/conv2d_51 bn
I0322 14:12:41.538240  8916 utils.py:47] yolo_conv_0/conv2d_52 bn
I0322 14:12:41.545222  8916 utils.py:47] yolo_conv_0/conv2d_53 bn
I0322 14:12:41.597058  8916 utils.py:47] yolo_conv_0/conv2d_54 bn
I0322 14:12:41.603073  8916 utils.py:47] yolo_conv_0/conv2d_55 bn
I0322 14:12:41.655930  8916 utils.py:47] yolo_conv_0/conv2d_56 bn
I0322 14:12:41.663909  8916 utils.py:47] yolo_output_0/conv2d_57 bn
I0322 14:12:41.713775  8916 utils.py:47] yolo_output_0/conv2d_58 bias
I0322 14:12:41.717764  8916 utils.py:47] yolo_conv_1/conv2d_59 bn
I0322 14:12:41.719759  8916 utils.py:47] yolo_conv_1/conv2d_60 bn
I0322 14:12:41.722752  8916 utils.py:47] yolo_conv_1/conv2d_61 bn
I0322 14:12:41.732725  8916 utils.py:47] yolo_conv_1/conv2d_62 bn
I0322 14:12:41.735716  8916 utils.py:47] yolo_conv_1/conv2d_63 bn
I0322 14:12:41.746687  8916 utils.py:47] yolo_conv_1/conv2d_64 bn
I0322 14:12:41.749615  8916 utils.py:47] yolo_output_1/conv2d_65 bn
I0322 14:12:41.760585  8916 utils.py:47] yolo_output_1/conv2d_66 bias
I0322 14:12:41.762580  8916 utils.py:47] yolo_conv_2/conv2d_67 bn
I0322 14:12:41.764575  8916 utils.py:47] yolo_conv_2/conv2d_68 bn
I0322 14:12:41.766757  8916 utils.py:47] yolo_conv_2/conv2d_69 bn
I0322 14:12:41.769778  8916 utils.py:47] yolo_conv_2/conv2d_70 bn
I0322 14:12:41.771775  8916 utils.py:47] yolo_conv_2/conv2d_71 bn
I0322 14:12:41.777057  8916 utils.py:47] yolo_conv_2/conv2d_72 bn
I0322 14:12:41.779037  8916 utils.py:47] yolo_output_2/conv2d_73 bn
I0322 14:12:41.782189  8916 utils.py:47] yolo_output_2/conv2d_74 bias
I0322 14:12:41.783190  8916 load_weights.py:22] weights loaded
2020-03-22 14:12:41.800300: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-03-22 14:12:43.102478: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows
Relying on driver to perform ptx compilation. This message will be only logged once.
2020-03-22 14:12:43.215831: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cublas64_100.dll'; dlerror: cublas64_100.dll not found
2020-03-22 14:12:43.220370: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_INTERNAL_ERROR
2020-03-22 14:12:43.224648: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
  File "load_weights.py", line 34, in <module>
    app.run(main)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "load_weights.py", line 25, in main
    output = yolo(img)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 708, in call
    convert_kwargs_to_constants=base_layer_utils.call_context().saving)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 860, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 708, in call
    convert_kwargs_to_constants=base_layer_utils.call_context().saving)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 860, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\keras\layers\convolutional.py", line 197, in call
    outputs = self._convolution_op(inputs, self.kernel)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 1134, in __call__
    return self.conv_op(inp, filter)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 639, in __call__
    return self.call(inp, filter)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 238, in __call__
    name=self.name)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 2010, in conv2d
    name=name)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 1031, in conv2d
    data_format=data_format, dilations=dilations, name=name, ctx=_ctx)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 1130, in conv2d_eager_fallback
    ctx=_ctx, name=name)
  File "C:\Users\rob26\anaconda3\envs\yolov3-gpu\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=25600, n=32, k=64 [Op:Conv2D]
rob26r commented 4 years ago

I found the solution and was able to run everything successfully. CUDA Toolkit 10.2 is missing certain dll files that older version of CUDA toolkit had. I found this link that has all of the missing dll files that you want save at this directory: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin

https://medium.com/@iitbguha/tensorflow-with-gpu-installation-made-easy-659f88c0309b