search

experiencor / keras-yolo3

Training and Detecting Objects with YOLO3
MIT License
1.6k stars 861 forks source link

tensor problem #186

Open vcvishal opened 5 years ago

vcvishal commented 5 years ago

Using TensorFlow backend. valid_annot_folder not exists. Spliting the trainining set. Seen labels: {'WBC': 372, 'RBC': 4155, 'Platelets': 361}

Given labels: ['Platelets', 'RBC', 'WBC']

Training on: ['Platelets', 'RBC', 'WBC']

2019-05-14 20:56:59.941932: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2019-05-14 20:57:00.898749: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62 pciBusID: 0000:01:00.0 totalMemory: 4.00GiB freeMemory: 3.30GiB 2019-05-14 20:57:00.907040: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0 2019-05-14 20:57:01.467095: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-05-14 20:57:01.471892: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0 2019-05-14 20:57:01.474685: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 2019-05-14 20:57:01.478907: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3030 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1) WARNING:tensorflow:From D:\miniconda\lib\site-packages\keras\backend\tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From D:\miniconda\lib\site-packages\keras\backend\tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead Epoch 1/103 resizing: 256 256 2019-05-14 20:58:51.568129: E T:\src\github\tensorflow\tensorflow\core\grappler\clusters\utils.cc:81] Failed to get device properties, error code: 30 Traceback (most recent call last): File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call return fn(*args) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero [[Node: replica_0/model_1/yolo_layer_1/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _class=["loc:@train...ad/Reshape"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](replica_0/model_1/conv_81/BiasAdd, replica_0/model_1/yolo_layer_1/concat)]] [[Node: training/Adam/gradients/replica_0/model_1/bnorm_84/batchnorm/mul_grad/Mul_1/_5057 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_36076...grad/Mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train.py", line 280, in main(args) File "train.py", line 257, in main max_queue_size = 8 File "D:\miniconda\lib\site-packages\keras\legacy\interfaces.py", line 87, in wrapper return func(*args, kwargs) File "D:\miniconda\lib\site-packages\keras\engine\training.py", line 2042, in fit_generator class_weight=class_weight) File "D:\miniconda\lib\site-packages\keras\engine\training.py", line 1762, in train_on_batch outputs = self.train_function(ins) File "D:\miniconda\lib\site-packages\keras\backend\tensorflow_backend.py", line 2273, in call self.session_kwargs) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 900, in run run_metadata_ptr) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run run_metadata) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero [[Node: replica_0/model_1/yolo_layer_1/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _class=["loc:@train...ad/Reshape"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](replica_0/model_1/conv_81/BiasAdd, replica_0/model_1/yolo_layer_1/concat)]] [[Node: training/Adam/gradients/replica_0/model_1/bnorm_84/batchnorm/mul_grad/Mul_1/_5057 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_36076...grad/Mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'replica_0/model_1/yolo_layer_1/Reshape', defined at: File "train.py", line 280, in main(args) File "train.py", line 242, in main class_scale = config['train']['class_scale'], File "train.py", line 157, in create_model train_model = multi_gpu_model(template_model, gpus=multi_gpu) File "C:\Users\vcvis\Desktop\keras-yolo3-master\utils\multi_gpu_model.py", line 48, in multi_gpu_model outputs = model(inputs) File "D:\miniconda\lib\site-packages\keras\engine\topology.py", line 602, in call output = self.call(inputs, kwargs) File "D:\miniconda\lib\site-packages\keras\engine\topology.py", line 2058, in call outputtensors, , _ = self.run_internal_graph(inputs, masks) File "D:\miniconda\lib\site-packages\keras\engine\topology.py", line 2220, in run_internal_graph output_tensors = _to_list(layer.call(computed_tensors, kwargs)) File "C:\Users\vcvis\Desktop\keras-yolo3-master\yolo.py", line 37, in call y_pred = tf.reshape(y_pred, tf.concat([tf.shape(y_pred)[:3], tf.constant([3, -1])], axis=0)) File "D:\miniconda\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 7323, in reshape "Reshape", tensor=tensor, shape=shape, name=name) File "D:\miniconda\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "D:\miniconda\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op op_def=op_def) File "D:\miniconda\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero [[Node: replica_0/model_1/yolo_layer_1/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _class=["loc:@train...ad/Reshape"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](replica_0/model_1/conv_81/BiasAdd, replica_0/model_1/yolo_layer_1/concat)]] [[Node: training/Adam/gradients/replica_0/model_1/bnorm_84/batchnorm/mul_grad/Mul_1/_5057 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_36076...grad/Mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

please help

abert1 commented 5 years ago

same problem here ! Anybody has an idea ?

kabrau commented 5 years ago

I think your dataset has a problem, but I do not know what, it look that the net does not reshape images or tensor. Try train with few samples only, for example, {'WBC': 10, 'RBC': 10, 'Platelets': 10}, maybe you can figure out the problem.

jxingm commented 5 years ago

Using TensorFlow backend. valid_annot_folder not exists. Spliting the trainining set. Seen labels: {'WBC': 372, 'RBC': 4155, 'Platelets': 361}

Given labels: ['Platelets', 'RBC', 'WBC']

Training on: ['Platelets', 'RBC', 'WBC']

2019-05-14 20:56:59.941932: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2019-05-14 20:57:00.898749: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62 pciBusID: 0000:01:00.0 totalMemory: 4.00GiB freeMemory: 3.30GiB 2019-05-14 20:57:00.907040: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0 2019-05-14 20:57:01.467095: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-05-14 20:57:01.471892: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0 2019-05-14 20:57:01.474685: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 2019-05-14 20:57:01.478907: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3030 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1) WARNING:tensorflow:From D:\miniconda\lib\site-packages\keras\backend\tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From D:\miniconda\lib\site-packages\keras\backend\tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead Epoch 1/103 resizing: 256 256 2019-05-14 20:58:51.568129: E T:\src\github\tensorflow\tensorflow\core\grappler\clusters\utils.cc:81] Failed to get device properties, error code: 30 Traceback (most recent call last): File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call return fn(*args) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero [[Node: replica_0/model_1/yolo_layer_1/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _class=["loc:@train...ad/Reshape"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](replica_0/model_1/conv_81/BiasAdd, replica_0/model_1/yolo_layer_1/concat)]] [[Node: training/Adam/gradients/replica_0/model_1/bnorm_84/batchnorm/mul_grad/Mul_1/_5057 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_36076...grad/Mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train.py", line 280, in main(args) File "train.py", line 257, in main max_queue_size = 8 File "D:\miniconda\lib\site-packages\keras\legacy\interfaces.py", line 87, in wrapper return func(*args, kwargs) File "D:\miniconda\lib\site-packages\keras\engine\training.py", line 2042, in fit_generator class_weight=class_weight) File "D:\miniconda\lib\site-packages\keras\engine\training.py", line 1762, in train_on_batch outputs = self.train_function(ins) File "D:\miniconda\lib\site-packages\keras\backend\tensorflow_backend.py", line 2273, in call self.session_kwargs) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 900, in run run_metadata_ptr) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run run_metadata) File "D:\miniconda\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero [[Node: replica_0/model_1/yolo_layer_1/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _class=["loc:@train...ad/Reshape"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](replica_0/model_1/conv_81/BiasAdd, replica_0/model_1/yolo_layer_1/concat)]] [[Node: training/Adam/gradients/replica_0/model_1/bnorm_84/batchnorm/mul_grad/Mul_1/_5057 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_36076...grad/Mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'replica_0/model_1/yolo_layer_1/Reshape', defined at: File "train.py", line 280, in main(args) File "train.py", line 242, in main class_scale = config['train']['class_scale'], File "train.py", line 157, in create_model train_model = multi_gpu_model(template_model, gpus=multi_gpu) File "C:\Users\vcvis\Desktop\keras-yolo3-master\utils\multi_gpu_model.py", line 48, in multi_gpu_model outputs = model(inputs) File "D:\miniconda\lib\site-packages\keras\engine\topology.py", line 602, in call output = self.call(inputs, kwargs) File "D:\miniconda\lib\site-packages\keras\engine\topology.py", line 2058, in call outputtensors, , _ = self.run_internal_graph(inputs, masks) File "D:\miniconda\lib\site-packages\keras\engine\topology.py", line 2220, in run_internal_graph output_tensors = _to_list(layer.call(computed_tensors, kwargs)) File "C:\Users\vcvis\Desktop\keras-yolo3-master\yolo.py", line 37, in call y_pred = tf.reshape(y_pred, tf.concat([tf.shape(y_pred)[:3], tf.constant([3, -1])], axis=0)) File "D:\miniconda\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 7323, in reshape "Reshape", tensor=tensor, shape=shape, name=name) File "D:\miniconda\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "D:\miniconda\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op op_def=op_def) File "D:\miniconda\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero [[Node: replica_0/model_1/yolo_layer_1/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _class=["loc:@train...ad/Reshape"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](replica_0/model_1/conv_81/BiasAdd, replica_0/model_1/yolo_layer_1/concat)]] [[Node: training/Adam/gradients/replica_0/model_1/bnorm_84/batchnorm/mul_grad/Mul_1/_5057 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_36076...grad/Mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

please help

Hi,I got the same error,have you solved it?? Thanks!