search

tensorflow / tensorboard

TensorFlow's Visualization Toolkit
Apache License 2.0
6.73k stars 1.66k forks source link

Images tab is not updating #1257

Open austinmw opened 6 years ago

austinmw commented 6 years ago

I'm running tensorboard to track an object detection model (ssd-mobilenet-v2) while running the object_detection/eval.py script. The issue I'm having is that everything seems to be updating properly with the exception of the IMAGES tab. I also don't see any way to select the iteration number within this tab. It's still showing a wall time of the first evaluation iteration from a few hours ago. All other tabs seem to be current.

nfelt commented 6 years ago

Could you include a screenshot and more detail on exactly what command you ran? If you can attach the tfevents file from the log directory you're using that would also help, and include any output printed by the script.

austinmw commented 6 years ago

Sure. wall time for scalars tab vs wall time for images tab

started tensorboard of course just with tensorboard --logdir=., ran eval with:

run_eval()
{
export CUDA_VISIBLE_DEVICES=3
python /root/training/eval.py --logtostderr --pipeline_config_path=/root/training/ssd_mobilenet_v2_coco.config --checkpoint_dir=/root/training/models/train --eval_dir=/root/training/models/eval --num_clones=1 --ps_tasks=1
unset CUDA_VISIBLE_DEVICES
}

eval.py hasn't been changed from default in tensorflow/models/research/object_detection/. Settings from ssd_mobilenet_v2_coco.config related to evaluation are:

eval_config: {
  num_examples: 500
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 1000
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: "/root/training/data/val.record"
  }
  label_map_path: "/root/training/data/label_map.pbtxt"
  shuffle: false
  num_readers: 1
}

Full terminal output so far from the eval.py script is:

/tensorflow/models/research/object_detection/utils/visualization_utils.py:25: UserWarning:
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was *originally* set to 'Qt5Agg' by the following code:
  File "/root/training/eval.py", line 50, in <module>
    from object_detection import evaluator
  File "/tensorflow/models/research/object_detection/evaluator.py", line 24, in <module>
    from object_detection import eval_util
  File "/tensorflow/models/research/object_detection/eval_util.py", line 35, in <module>
    from object_detection.metrics import coco_evaluation
  File "/tensorflow/models/research/object_detection/metrics/coco_evaluation.py", line 20, in <module>
    from object_detection.metrics import coco_tools
  File "/tensorflow/models/research/object_detection/metrics/coco_tools.py", line 47, in <module>
    from pycocotools import coco
  File "/tensorflow/models/research/pycocotools/coco.py", line 49, in <module>
    import matplotlib.pyplot as plt
  File "/opt/conda/lib/python3.6/site-packages/matplotlib/pyplot.py", line 71, in <module>
    from matplotlib.backends import pylab_setup
  File "/opt/conda/lib/python3.6/site-packages/matplotlib/backends/__init__.py", line 16, in <module>
    line for line in traceback.format_stack()

  import matplotlib; matplotlib.use('Agg')  # pylint: disable=multiple-statements
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
2018-06-18 14:55:25.514995: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-06-18 14:55:25.830988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:0a:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2018-06-18 14:55:25.831030: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 14:55:26.104078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 14:55:26.104118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 14:55:26.104128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 14:55:26.104395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-0
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-0
2018-06-18 15:05:25.749344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 15:05:25.749462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 15:05:25.749472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 15:05:25.749481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 15:05:25.749654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-3520
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-3520
2018-06-18 15:15:25.829870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 15:15:25.829920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 15:15:25.829940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 15:15:25.829963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 15:15:25.830123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-7218
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-7218
2018-06-18 15:25:25.840161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 15:25:25.840212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 15:25:25.840275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 15:25:25.840290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 15:25:25.840451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-10934
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-10934
2018-06-18 15:35:25.933045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 15:35:25.933097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 15:35:25.933109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 15:35:25.933116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 15:35:25.933310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-14542
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-14542
2018-06-18 15:45:26.022446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 15:45:26.022496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 15:45:26.022508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 15:45:26.022516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 15:45:26.022688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-18247
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-18247
2018-06-18 15:55:26.054286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 15:55:26.054403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 15:55:26.054419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 15:55:26.054439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 15:55:26.054605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-21872
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-21872
2018-06-18 16:05:26.142559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 16:05:26.142664: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 16:05:26.142678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 16:05:26.142687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 16:05:26.142839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-25595
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-25595
2018-06-18 16:15:26.322853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 16:15:26.322963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 16:15:26.322975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 16:15:26.322984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 16:15:26.323189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-29325
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-29325
2018-06-18 16:25:26.455067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 16:25:26.455139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 16:25:26.455161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 16:25:26.455174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 16:25:26.455409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-33037
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-33037
2018-06-18 16:35:26.626782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 16:35:26.626835: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 16:35:26.626846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 16:35:26.626855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 16:35:26.627012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-36768
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-36768
2018-06-18 16:45:26.729614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 16:45:26.729670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 16:45:26.729681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 16:45:26.729698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 16:45:26.729859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-40489
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-40489
2018-06-18 16:55:26.890692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 16:55:26.890767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 16:55:26.890785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 16:55:26.890799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 16:55:26.891681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-44192
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-44192
2018-06-18 17:05:27.056794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 17:05:27.056840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 17:05:27.056849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 17:05:27.056855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 17:05:27.057003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-47912
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-47912
2018-06-18 17:15:27.237190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 17:15:27.237283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 17:15:27.237304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 17:15:27.237321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 17:15:27.237546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-50722
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-50722
2018-06-18 17:25:27.370016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 17:25:27.370067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 17:25:27.370079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 17:25:27.370090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 17:25:27.373029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-54522
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-54522
2018-06-18 17:35:27.447574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 17:35:27.447617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 17:35:27.447628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 17:35:27.447641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 17:35:27.447785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-58335
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-58335
2018-06-18 17:45:27.586037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 17:45:27.586080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 17:45:27.586093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 17:45:27.586105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 17:45:27.586305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-62074
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-62074
2018-06-18 17:55:27.749948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 17:55:27.749994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 17:55:27.750005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 17:55:27.750014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 17:55:27.750180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-65787
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-65787
2018-06-18 18:05:27.842047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 18:05:27.842132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 18:05:27.842144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 18:05:27.842150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 18:05:27.842327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-69564
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-69564
2018-06-18 18:15:27.931603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 18:15:27.931701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 18:15:27.931713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 18:15:27.931721: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 18:15:27.931888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-73382
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-73382
2018-06-18 18:25:28.056631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 18:25:28.056726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 18:25:28.056737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 18:25:28.056745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 18:25:28.056911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-77206
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-77206
2018-06-18 18:35:28.222850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 18:35:28.222907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 18:35:28.222921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 18:35:28.222940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 18:35:28.223125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-81000
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-81000
2018-06-18 18:45:28.402418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 18:45:28.402480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 18:45:28.402492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 18:45:28.402503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 18:45:28.402673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-84621
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-84621
2018-06-18 18:55:28.517022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 18:55:28.517077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 18:55:28.517095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 18:55:28.517106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 18:55:28.517236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-88199
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-88199
2018-06-18 19:05:28.620397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 19:05:28.620497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 19:05:28.620512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 19:05:28.620521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 19:05:28.620682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-91924
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-91924
2018-06-18 19:15:28.784978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-18 19:15:28.785026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-18 19:15:28.785036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-18 19:15:28.785045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-18 19:15:28.785202: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-95615
INFO:tensorflow:Restoring parameters from /root/training/models/train/model.ckpt-95615

I'll pull the events file off my server and upload shortly, but do there happen to be any settings I configured improperly in order to update images?

nfelt commented 6 years ago

I'm not that familiar with the object detection models, but what's your setting for eval_config.num_visualizations?

https://github.com/tensorflow/models/blob/324d6dc3b52d7219d4ef48b1e4b7f9e4086f11fd/research/object_detection/protos/eval.proto#L8

austinmw commented 6 years ago

I think it's set to the default of 10 currently. I was under the impression that it would show the first ten every time (unless I set shuffle=true), but should at least update after each new checkpoint evaluation, no?

nfelt commented 6 years ago

I'm not sure - as I said, not that familiar with the TF models code. Did you get ahold of the events file? That's going to be the easiest way for me to provide any more information.