after run model_main.py

[786694836]

It show me this :

creating index... index created! creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=39.81s). Accumulating evaluation results... DONE (t=11.59s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.334 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.548 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.350 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.014 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.110 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.455 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.357 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.490 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.514 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.070 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.311 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.638 2018-11-07 04:40:20.876549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0 2018-11-07 04:40:20.876662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-11-07 04:40:20.876675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] 0 2018-11-07 04:40:20.876682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0: N 2018-11-07 04:40:20.877056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7942 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:03:00.0, compute capability: 6.0) WARNING:tensorflow:From /home-ex/tclsz/miniconda3/envs/tensorflow-abb/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py:1018: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version. Instructions for updating: Pass your op to the equivalent parameter main_op instead. W1107 04:40:21.531219 140403135842112 tf_logging.py:125] From /home-ex/tclsz/miniconda3/envs/tensorflow-abb/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py:1018: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version. Instructions for updating: Pass your op to the equivalent parameter main_op instead.

and then It stopped.

I don't know what it means.

So who can tell me what the right method to train the ssdlite model that can show me the steps and the loss ?

[786694836]

and I have an other question, how can I look the train massage like loss = ?

[WendieSun]

I'm having this problem too, only saw Average Precision and Average Recall, it doesn't show me anything about loss..?

[saleem-hadad]

same here

[zubairahmed-ai]

Under models/research/object_detection/model_main.py after imports add the following

tf.logging.set_verbosity(tf.logging.INFO)

[saleem-hadad]

thank you @zubairahmed-ai it was working actually but without console output, after a while, the files appeared in the output directory

[zubairahmed-ai]

@saleem-hadad wait a minute, do you actually see checkpoint files being saved in the output directory?

[saleem-hadad]

yup

[zubairahmed-ai]

I literally just posted this, do you have a solution for this issue? https://github.com/tensorflow/models/issues/5810

[saleem-hadad]

I'll check

[saleem-hadad]

In my case, I run it on google cloud with 4 GPUs that why the result came out very fast

[zubairahmed-ai]

Oh wow, 12+ hours and my model is still training locally and it's going to take another couple of hours

[saleem-hadad]

Yeah that's normal 🐢

[navdeep1604]

I have run model and Now not able to interpret results and want to know, I didnot see any checkpoint file saved. Could you help what needs to be done once model run, I ran for number_steps=2 intially and now how to fine tune and use which model for prediction at run time.

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 WARNING:tensorflow:From C:\Users\navdeep.e.singh\AppData\Roaming\Python\Python35\site-packages\tensorflow\python\estimator\estimator.py:1044: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version. Instructions for updating: Pass your op to the equivalent parameter main_op instead. WARNING:tensorflow:From C:\Users\navdeep.e.singh\AppData\Roaming\Python\Python35\site-packages\tensorflow\python\estimator\estimator.py:1044: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version. Instructions for updating: Pass your op to the equivalent parameter main_op instead.

[xqxqwps]

Under models/research/object_detection/model_main.py after imports add the following

tf.logging.set_verbosity(tf.logging.INFO)

I do it,but train only once and stop

[xqxqwps]

train data size have to be divisible by batch size.it works,but I dont know why.If you get it ,please tell me.

[ashkanee]

I have the same problem here, the training stops after a while without any clear error message. I just get a warning as follows before the run stops:

WARNING:tensorflow:From /home/ashkan/anaconda3/envs/py365/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py:1018: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version.
Instructions for updating:
Pass your op to the equivalent parameter main_op instead.

[benni-t]

did anyone solve the problem???

I have the same problem here, the training stops after a while without any clear error message. I just get a warning as follows before the run stops:

WARNING:tensorflow:From /home/ashkan/anaconda3/envs/py365/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py:1018: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version.
Instructions for updating:
Pass your op to the equivalent parameter main_op instead.

[abderrazzak555]

Please, how can I calculate the MAP for these results?

INFO:tensorflow:Restoring parameters from test_image1/model.ckpt-50000 INFO:tensorflow:Restoring parameters from test_image1/model.ckpt-50000 INFO:tensorflow:Running local_init_op. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Performing evaluation on 4 images. INFO:tensorflow:Performing evaluation on 4 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... INFO:tensorflow:Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.00s) INFO:tensorflow:DONE (t=0.00s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=0.06s). Accumulating evaluation results... DONE (t=0.01s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.211 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.380 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.187 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.003 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.233 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.485 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.067 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.241 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.261 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.020 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.283 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.570 INFO:tensorflow:Finished evaluation at 2019-10-28-14:06:58 INFO:tensorflow:Finished evaluation at 2019-10-28-14:06:58 INFO:tensorflow:Saving dict for global step 50000: DetectionBoxes_Precision/mAP = 0.21072435, DetectionBoxes_Precision/mAP (large) = 0.485231, DetectionBoxes_Precision/mAP (medium) = 0.23299623, DetectionBoxes_Precision/mAP (small) = 0.0032204273, DetectionBoxes_Precision/mAP@.50IOU = 0.38036388, DetectionBoxes_Precision/mAP@.75IOU = 0.1866721, DetectionBoxes_Recall/AR@1 = 0.06658986, DetectionBoxes_Recall/AR@10 = 0.24055299, DetectionBoxes_Recall/AR@100 = 0.26059908, DetectionBoxes_Recall/AR@100 (large) = 0.57045454, DetectionBoxes_Recall/AR@100 (medium) = 0.2825, DetectionBoxes_Recall/AR@100 (small) = 0.02, Loss/BoxClassifierLoss/classification_loss = 0.5046802, Loss/BoxClassifierLoss/localization_loss = 0.3415953, Loss/RPNLoss/localization_loss = 0.54075974, Loss/RPNLoss/objectness_loss = 0.46926486, Loss/total_loss = 1.8563001, global_step = 50000, learning_rate = 0.0002, loss = 1.8563001 INFO:tensorflow:Saving dict for global step 50000: DetectionBoxes_Precision/mAP = 0.21072435, DetectionBoxes_Precision/mAP (large) = 0.485231, DetectionBoxes_Precision/mAP (medium) = 0.23299623, DetectionBoxes_Precision/mAP (small) = 0.0032204273, DetectionBoxes_Precision/mAP@.50IOU = 0.38036388, DetectionBoxes_Precision/mAP@.75IOU = 0.1866721, DetectionBoxes_Recall/AR@1 = 0.06658986, DetectionBoxes_Recall/AR@10 = 0.24055299, DetectionBoxes_Recall/AR@100 = 0.26059908, DetectionBoxes_Recall/AR@100 (large) = 0.57045454, DetectionBoxes_Recall/AR@100 (medium) = 0.2825, DetectionBoxes_Recall/AR@100 (small) = 0.02, Loss/BoxClassifierLoss/classification_loss = 0.5046802, Loss/BoxClassifierLoss/localization_loss = 0.3415953, Loss/RPNLoss/localization_loss = 0.54075974, Loss/RPNLoss/objectness_loss = 0.46926486, Loss/total_loss = 1.8563001, global_step = 50000, learning_rate = 0.0002, loss = 1.8563001 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 50000: test_image1/model.ckpt-50000 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 50000: test_image1/model.ckpt-50000 INFO:tensorflow:Performing the final export in the end of training. INFO:tensorflow:Performing the final export in the end of training.

[nanshihui]

any volunteer?

[neilyoung]

Same issue, three years later. Thanks Obama :)