Closed MonkeyWithAComputer closed 5 years ago
MohammadMoradi
commented
6 years ago I have the same problem with the sample named _ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14sync.config. I just change the path and batch size and other things are the same.
MichaelX99
commented
6 years ago One line 703 of utils/visualization_utils.py change cdf_plot=tf.py_func(cdf_plot, [values], tf.uint8) to cdf_plot=tf.py_func(cdf_plot, [values], tf.float32)
this will eliminate the errors, or at least it did for me
MonkeyWithAComputer
commented
6 years ago @MichaelX99 That doesn't seem to change anything for me. I still get the main thread is not in main loop issue.
MonkeyWithAComputer
commented
6 years ago If it helps, there is no issue with training ssd_inception_v2_coco_2018_01_28, ssd_mobilenet_v2_coco_2018_03_29, or faster_rcnn_resnet101_coco_2018_01_28. However with ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03 and ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03 there is a main thread is not in main loop error. @bitfort
kakpple
commented
6 years ago Hey, guys, I have the same issue, any solutions? faster_rcnn runs fine by the way.
And training log is not showing by default, any solution for this?
kakpple
commented
6 years ago For work around solution, commenting out the following two lines from visualization_utils.py 703 // cdf_plot = tf.py_func(cdf_plot, [values], tf.uint8) 704 // tf.summary.image(name, cdf_plot)
Since this is image summary visualization function, will have no effect on training.
MonkeyWithAComputer
commented
6 years ago For work around solution, commenting out the following two lines from visualization_utils.py 703 // cdf_plot = tf.py_func(cdf_plot, [values], tf.uint8) 704 // tf.summary.image(name, cdf_plot)
Since this is image summary visualization function, will have no effect on training.
This worked, Thank you @kakpple
willSapgreen
commented
6 years ago For work around solution, commenting out the following two lines from visualization_utils.py 703 // cdf_plot = tf.py_func(cdf_plot, [values], tf.uint8) 704 // tf.summary.image(name, cdf_plot)
This worked for me. The git repo is on origin/master branch,
Merge pull request #4885 from norouzi/master
xtianhb
commented
6 years ago It worked for me, solved that error.
wt-huang
commented
5 years ago Closing as this is resolved
alon1samuel
commented
5 years ago For work around solution, commenting out the following two lines from visualization_utils.py 703 // cdf_plot = tf.py_func(cdf_plot, [values], tf.uint8) 704 // tf.summary.image(name, cdf_plot)
Since this is image summary visualization function, will have no effect on training.
This worked, Thank you @kakpple
Thanks @kakpple , it worked for me as well. :)
System information
python object_detection/model_main.py -pipeline_config_path=/home/dev/ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync.config -model_dir=/home/dev/train/Describe the problem
When I run the command above I get an error that the main thread is not in main loop error. Any idea what is causing this?
Source code / logs
`dev@dev-pc:~/models/research$ python object_detection/model_main.py -pipeline_config_path=/home/dev/ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync.config -model_dir=/home/dev/train/ /home/dev/models/research/object_detection/utils/visualization_utils.py:25: UserWarning: This call to matplotlib.use() has no effect because the backend has already been chosen; matplotlib.use() must be called before pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time.
The backend was originally set to 'TkAgg' by the following code: File "object_detection/model_main.py", line 26, in
from object_detection import model_lib
File "/home/dev/models/research/object_detection/model_lib.py", line 26, in
from object_detection import eval_util
File "/home/dev/models/research/object_detection/eval_util.py", line 28, in
from object_detection.metrics import coco_evaluation
File "/home/dev/models/research/object_detection/metrics/coco_evaluation.py", line 20, in
from object_detection.metrics import coco_tools
File "/home/dev/models/research/object_detection/metrics/coco_tools.py", line 47, in
from pycocotools import coco
File "/home/dev/models/research/pycocotools/coco.py", line 49, in
import matplotlib.pyplot as plt
File "/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 71, in
from matplotlib.backends import pylab_setup
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/init.py", line 16, in
line for line in traceback.format_stack()
import matplotlib; matplotlib.use('Agg') # pylint: disable=multiple-statements WARNING:tensorflow:Estimator's model_fn (<function model_fn at 0x7fe5818138c0>) includes params argument, but params are not passed to Estimator. WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_1/biases] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_2/biases] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_3/biases] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/weights] is not available in checkpoint 2018-07-16 18:31:06.137149: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2018-07-16 18:31:06.583642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: name: Graphics Device major: 7 minor: 0 memoryClockRate(GHz): 1.455 pciBusID: 0000:05:00.0 totalMemory: 11.77GiB freeMemory: 11.08GiB 2018-07-16 18:31:06.859931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 1 with properties: name: Graphics Device major: 7 minor: 0 memoryClockRate(GHz): 1.455 pciBusID: 0000:06:00.0 totalMemory: 11.78GiB freeMemory: 11.36GiB 2018-07-16 18:31:07.147017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 2 with properties: name: Graphics Device major: 7 minor: 0 memoryClockRate(GHz): 1.455 pciBusID: 0000:09:00.0 totalMemory: 11.78GiB freeMemory: 11.36GiB 2018-07-16 18:31:07.430797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 3 with properties: name: Graphics Device major: 7 minor: 0 memoryClockRate(GHz): 1.455 pciBusID: 0000:0a:00.0 totalMemory: 11.78GiB freeMemory: 11.36GiB 2018-07-16 18:31:07.437398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0, 1, 2, 3 2018-07-16 18:31:08.245701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-07-16 18:31:08.245752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 1 2 3 2018-07-16 18:31:08.245758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N Y Y Y 2018-07-16 18:31:08.245762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1: Y N Y Y 2018-07-16 18:31:08.245766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 2: Y Y N Y 2018-07-16 18:31:08.245769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 3: Y Y Y N 2018-07-16 18:31:08.246583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10706 MB memory) -> physical GPU (device: 0, name: Graphics Device, pci bus id: 0000:05:00.0, compute capability: 7.0) 2018-07-16 18:31:08.347404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10976 MB memory) -> physical GPU (device: 1, name: Graphics Device, pci bus id: 0000:06:00.0, compute capability: 7.0) 2018-07-16 18:31:08.450352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10976 MB memory) -> physical GPU (device: 2, name: Graphics Device, pci bus id: 0000:09:00.0, compute capability: 7.0) 2018-07-16 18:31:08.553383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10976 MB memory) -> physical GPU (device: 3, name: Graphics Device, pci bus id: 0000:0a:00.0, compute capability: 7.0) 2018-07-16 18:31:22.596064: W tensorflow/core/framework/op_kernel.cc:1306] Unknown: exceptions.RuntimeError: main thread is not in main loop Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/script_ops.py", line 157, in call ret = func(*args)
File "/home/dev/models/research/object_detection/utils/visualization_utils.py", line 693, in cdf_plot fig = plt.figure(frameon=False)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 548, in figure **kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager return cls.new_figure_manager_given_figure(num, fig)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/_backend_tk.py", line 1053, in new_figure_manager_given_figure icon_img = Tk.PhotoImage(file=icon_fname)
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 3370, in init Image.init(self, 'photo', name, cnf, master, **kw)
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 3324, in init self.tk.call(('image', 'create', imgtype, name,) + options)
RuntimeError: main thread is not in main loop
Traceback (most recent call last): File "object_detection/model_main.py", line 101, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "object_detection/model_main.py", line 97, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate
executor.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 518, in run
self.run_local()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 650, in run_local
hooks=train_hooks)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 859, in _train_model_default
saving_listeners)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1059, in _train_with_estimatorspec
, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 567, in run
run_metadata=run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1043, in run
run_metadata=run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1134, in run
raise six.reraise(original_exc_info)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1119, in run
return self._sess.run(args, *kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1191, in run
run_metadata=run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 971, in run
return self._sess.run(args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: exceptions.RuntimeError: main thread is not in main loop
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/script_ops.py", line 157, in call ret = func(*args)
File "/home/dev/models/research/object_detection/utils/visualization_utils.py", line 693, in cdf_plot fig = plt.figure(frameon=False)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 548, in figure **kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager return cls.new_figure_manager_given_figure(num, fig)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/_backend_tk.py", line 1053, in new_figure_manager_given_figure icon_img = Tk.PhotoImage(file=icon_fname)
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 3370, in init Image.init(self, 'photo', name, cnf, master, **kw)
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 3324, in init self.tk.call(('image', 'create', imgtype, name,) + options)
RuntimeError: main thread is not in main loop
Caused by op u'Loss/PyFunc_1', defined at: File "object_detection/model_main.py", line 101, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "object_detection/model_main.py", line 97, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate
executor.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 518, in run
self.run_local()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 650, in run_local
hooks=train_hooks)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 856, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 831, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/home/dev/models/research/object_detection/model_lib.py", line 281, in model_fn
prediction_dict, features[fields.InputDataFields.true_image_shape])
File "/home/dev/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 597, in loss
flattened_class_ids, flattened_classification_losses)
File "/home/dev/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 660, in _summarize_anchor_classification_loss
'NegativeAnchorLossCDF')
File "/home/dev/models/research/object_detection/utils/visualization_utils.py", line 703, in add_cdf_image_summary
cdf_plot = tf.py_func(cdf_plot, [values], tf.uint8)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/script_ops.py", line 344, in py_func
func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/script_ops.py", line 241, in _internal_py_func
input=inp, token=token, Tout=Tout, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_script_ops.py", line 128, in py_func
"PyFunc", input=input, token=token, Tout=Tout, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
UnknownError (see above for traceback): exceptions.RuntimeError: main thread is not in main loop Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/script_ops.py", line 157, in call ret = func(*args)
File "/home/dev/models/research/object_detection/utils/visualization_utils.py", line 693, in cdf_plot fig = plt.figure(frameon=False)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 548, in figure **kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager return cls.new_figure_manager_given_figure(num, fig)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/_backend_tk.py", line 1053, in new_figure_manager_given_figure icon_img = Tk.PhotoImage(file=icon_fname)
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 3370, in init Image.init(self, 'photo', name, cnf, master, **kw)
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 3324, in init self.tk.call(('image', 'create', imgtype, name,) + options)
RuntimeError: main thread is not in main loop
Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs func(*targs, *kargs) File "/usr/local/lib/python2.7/dist-packages/matplotlib/_pylab_helpers.py", line 78, in destroy_all manager.destroy() File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/_backend_tk.py", line 558, in destroy self.window.destroy() File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1862, in destroy for c in self.children.values(): c.destroy() File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 2100, in destroy self.tk.call('destroy', self._w) RuntimeError: main thread is not in main loop Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs func(targs, **kargs) File "/usr/local/lib/python2.7/dist-packages/matplotlib/_pylab_helpers.py", line 78, in destroy_all manager.destroy() File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/_backend_tk.py", line 558, in destroy self.window.destroy() File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1862, in destroy for c in self.children.values(): c.destroy() File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 2100, in destroy self.tk.call('destroy', self._w) RuntimeError: main thread is not in main loop`