training issue..... - Githubissues

INFO:tensorflow:using moving average in training, with decay = 0.999900 INFO:tensorflow:Ignoring --checkpoint_path because a checkpoint already exists in /home/longyan/OCR/ocr-data/icdar2015/log/ 2019-04-10 09:57:09.416707: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2019-04-10 09:57:09.416730: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2019-04-10 09:57:09.416754: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2019-04-10 09:57:09.416760: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2019-04-10 09:57:09.416786: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. 2019-04-10 09:57:09.488480: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-04-10 09:57:09.488817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: name: GeForce GTX 1060 with Max-Q Design major: 6 minor: 1 memoryClockRate (GHz) 1.3415 pciBusID 0000:01:00.0 Total memory: 5.94GiB Free memory: 5.51GiB 2019-04-10 09:57:09.488851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 2019-04-10 09:57:09.488858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0: Y 2019-04-10 09:57:09.488884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 with Max-Q Design, pci bus id: 0000:01:00.0) INFO:tensorflow:Restoring parameters from /home/longyan/OCR/ocr-data/icdar2015/log/model.ckpt-0 2019-04-10 09:57:09.630815: I tensorflow/core/common_runtime/simple_placer.cc:697] Ignoring device specification /device:GPU:0 for node 'clone_0/fifo_queue_Dequeue' because the input edge from 'icdar2015_prefetch_queue/prefetch_queue/fifo_queue' is a reference connection and already has a device field set to /device:CPU:0 INFO:tensorflow:Starting Session. INFO:tensorflow:Saving checkpoint to path /home/longyan/OCR/ocr-data/icdar2015/log/model.ckpt INFO:tensorflow:Starting Queues. INFO:tensorflow:global_step/sec: 0 2019-04-10 09:57:11.990542: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (156,-258,181,-211) is completely outside the image and will not be drawn. 2019-04-10 09:57:12.112359: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (156,-258,181,-211) is completely outside the image and will not be drawn. INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Incompatible shapes: [10] vs. [0] [[Node: ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/LogicalAnd = LogicalAnd[_device="/job:localhost/replica:0/task:0/cpu:0"](ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/Less, ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/Equal)]] INFO:tensorflow:Caught OutOfRangeError. Stopping Training. INFO:tensorflow:Finished training! Saving model to disk. 2019-04-10 09:57:13.203005: W tensorflow/core/framework/op_kernel.cc:1192] Unknown: exceptions.AssertionError: Traceback (most recent call last): File "train_pixel_link.py", line 293, in tf.app.run() File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "train_pixel_link.py", line 289, in main train(train_op) File "train_pixel_link.py", line 278, in train session_config = sess_config File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 767, in train sv.stop(threads, close_summary_writer=True) File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 792, in stop stop_grace_period_secs=self._stop_grace_secs) File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join six.reraise(*self._exc_info_to_raise) File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run enqueue_callable() File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1235, in _single_operation_run target_list_as_strings, status, None) File "/home/longyan/anaconda3/envs/py27/lib/python2.7/contextlib.py", line 24, in exit self.gen.next() File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [10] vs. [0] [[Node: ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/LogicalAnd = LogicalAnd[_device="/job:localhost/replica:0/task:0/cpu:0"](ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/Less, ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/Equal)]]

ZJULearning / pixel_link

training issue..... #122