Tensorflow realtime_object_detection on Jetson Xavier/TX2/TX1, PC

About this repository

forked from GustavZ/realtime_object_detection: https://github.com/GustavZ/realtime_object_detection
And focused on model split technique of ssd_mobilenet_v1.

Download model from here: tf1_detection_model_zoo

wget http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz

and here: TensorFlow DeepLab Model Zoo

wget http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_train_aug_2018_01_29.tar.gz

Support models

• TensorRT -> model_type: 'trt_v1'
  Requirements: https://github.com/NVIDIA-Jetson/tf_trt_models
  

• Faster R-CNN: PC/Xavier only
  faster_rcnn_nas_coco_2018_01_28 occurred Out Of Memory on my PC.
  Other Faster R-CNN has not checked yet.
  

• Mask R-CNN: PC/Xavier only
  Removed split_model.
  Add worker_threads for parallel detection. A little bit fast, maybe.
  

• DeepLab V3: PC/Xavier only
  

See also:

• https://github.com/tensorflow/models/issues/3270
• https://devblogs.nvidia.com/tensorrt-integration-speeds-tensorflow-inference/

Getting Started:

• login Jetson TX2. Desktop login or ssh remote login. ssh -C -Y ubuntu@xxx.xxx.xxx.xxx
• copy .config.yml to config.yml cp .config.yml config.yml
• edit config.yml for your environment. (Ex. camera_input: 0 # for PC)
• run python run_stream.py realtime object detection from webcam
• or run python run_video.py realtime object detection from movie file
• or run python run_image.py realtime object detection from image file
• wait a few minutes.
• Multi-Threading is better performance than Multi-Processing. Multi-Processing bottleneck is interprocess communication.
  

Requirements:

pip install --upgrade pyyaml

Also, OpenCV >= 3.1 and Tensorflow >= 1.4 (1.6 is good)

config.yml

Image

with run_image.py
Please create 'images' directory and put image files.(jpeg,jpg,png)
Subdirectories can also be used.

image_input: 'images'       # input image dir

Movie

with run_video.py

movie_input: 'input.mp4'    # mp4 or avi. Movie file.

Camera

with run_stream.py
This is OpenCV argument.

• USB Webcam on PC/Xavier

  camera_input: 0

• USB Webcam on TX2

  camera_input: 1

• Onboard camera on Xavier (with TX2 onboard camera)

  camera_input: "nvarguscamerasrc ! video/x-raw(memory:NVMM), width=1280, height=720,format=NV12, framerate=120/1 ! nvvidconv ! video/x-raw,format=I420 ! videoflip method=rotate-180 ! appsink"

• Onboard camera on TX2

  camera_input: "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"

Save to file

• Movie (run_stream.py or run_video.py)
  Save detection frame to movie file. (./output_movie/output_unixtime.avi)
  Requires a lot of disk space.
• Image (run_image.py)
  Save detection image to image file. (./output_image/PATH_TO_FILE/filename.jpg)
  Normally, this output image file is the same width x height and format as input images.
  But if run with MASK R-CNN, output file size is resized by width and height.

  save_to_file: True

Without Visualization

I do not know why, but in TX2 force_gpu_compatible: True it will be faster.

• on TX2

  force_gpu_compatible: True
  visualize: False

• on PC

  force_gpu_compatible: False
  visualize: False

With Visualization

Visualization is heavy. Visualization FPS possible to limit.
Display FPS: Detection FPS.

• default is with Single-Processing and show every frames.

  visualize: True
  vis_worker: False
  max_vis_fps: 0
  vis_text: True

• Visualization FPS limit with Single-Processing

  visualize: True
  vis_worker: False
  max_vis_fps: 30
  vis_text: True

• Visualization FPS limit with Multi-Processing
  This is good to use with save_to_file: True.

  visualize: True
  vis_worker: True
  max_vis_fps: 30
  vis_text: True

• Model type

  model_type: 'nms_v2'

  The difference between 'nms_v1' and 'nms_v2' is BatchMultiClassNonMaxSuppression inputs.
  model_type: trt_v1 is somewhat special. See config.yml.
  

# ssd_mobilenet_v1_coco_2018_01_28
model_type: 'nms_v2'
model_path: 'models/ssd_mobilenet_v1_coco_2018_01_28/frozen_inference_graph.pb'
label_path: 'models/labels/mscoco_label_map.pbtxt'
num_classes: 90

• Splite shape
  split_shape: 1917
  ExpandDims_1's shape. Ex:
  

See also: Learn Split Model

• TensorRT split/non-split both support. Need Tensorflow with TensorRT support. (r1.9 has bug. I use r1.8/r1.10.1 for pc)

  model_type: 'trt_v1'
  precision_model: 'FP32'     # 'FP32', 'FP16', 'INT8'
  model: 'ssd_inception_v2_coco_2018_01_28'
  label_path: 'models/labels/mscoco_label_map.pbtxt'
  num_classes: 90

Console Log

FPS:25.8  Frames:130 Seconds: 5.04248   | 1FRAME total: 0.11910   cap: 0.00013   gpu: 0.03837   cpu: 0.02768   lost: 0.05293   send: 0.03834   | VFPS:25.4  VFrames:128 VDrops: 1 

FPS: detection fps. average fps of fps_interval (5sec).
Frames: detection frames in fps_interval.
Seconds: fps_interval running time.

1FRAME
total: 1 frame's processing time. 0.1 means delay and 10 fps if it is single-threading(split_model: False). In multi-threading(split_model: True), this value means delay.
cap: time of capture camera image and transform for model input.
gpu: sess.run() time of gpu part.
cpu: sess.run() time of cpu part.
lost: time of overhead, something sleep etc.
send: time of multi-processing queue, block and pipe time.

VFPS: visualization fps.
VFrames: visualization frames in fps_interval.
VDrops: When multi-processing visualization is bottleneck, drops.

Updates:

• Support Xavier onboard camera. (with TX2 onboard camera)

• Add parallel detection for Mask R-CNN.

• Remove split from Mask R-CNN.

• Support DeepLab V3 models. model_type: deeplab_v3

• Add image input.

• Rename config.yml parameter name from save_to_movie to save_to_file.

• support Faster R-CNN models.

• Add max_frame: 0 for no exit with visualize: False.

• support ssd_mobilenet_v1 11 Jun, 2017 model.

• Add from movie.

• Add save_to_movie.

• BETA: Support MASK R-CNN models.

• Always split GPU/CPU device.

• Support SSD 2018_07_03 models.

• Support TensorRT Optimization. : Need TensorRT, Tensorflow with TensorRT.

• Support ssd_mobilenet_v2, ssdlite_mobilenet_v2 and ssd_inception_v2_coco. : Download model from here: detection_model_zoo

• Add Multi-Processing visualization. : Detection and visualization are asynchronous.

• Drop unused files.

• Add force_gpu_compatible option. : ssd_mobilenet_v1_coco 34.5 FPS without vizualization 1280x720 on TX2.

• Multi-Processing version corresponds to python 3.6 and python 2.7.

• Launch speed up. : Improve startup time from 90sec to 78sec.

• Add time details. : To understand the processing time well.

• Separate split and non-split code. : Remove unused session from split code.

• Remove Session from load frozen graph. : Reduction of memory usage.

• Flexible sleep_interval. : Maybe speed up on high performance PC.

• FPS separate to multi-processing. : Speed up.

• FPS streaming calculation. : Flat fps.

• FPS is average of fps_interval. : Flat fps.

• FPS updates every 0.2 sec. : Flat fps.

• solve: Multiple session cannot launch problem. tensorflow.python.framework.errors_impl.InternalError: Failed to create session.

My Setup:

• PC
  • CPU: i7-8700 3.20GHz 6-core 12-threads
  • GPU: NVIDIA GTX1060 6GB
  • MEMORY: 32GB
  • Ubuntu 16.04
  • docker-ce
  • nvidia-docker
    • nvidia/cuda
    • Pyton 2.7.12/OpenCV 3.4.1/Tensorflow 1.6.1
    • Pyton 3.6.5/OpenCV 3.4.1/Tensorflow 1.6.1
• Jetson Xavier
  • JetPack 4.0 Developer Preview
  • Python 2.7/OpenCV 3.3.1/Tensorflow 1.6.1
  • Python 2.7/OpenCV 3.3.1/Tensorflow 1.10.1 (slow)
  • JetPack 4.1.1 Developer Preview
  • Python 3.6.7/OpenCV 3.4.1/Tensorflow 1.10.1 (seems fast. I changed opencv build options.)
• Jetson TX2
  • JetPack 3.2/3.2.1
  • Python 3.6
  • OpenCV 3.4.1/Tensorflow 1.6.0
  • OpenCV 3.4.1/Tensorflow 1.6.1
  • OpenCV 3.4.1/Tensorflow 1.7.0 (slow)
  • OpenCV 3.4.1/Tensorflow 1.7.1 (slow)
  • OpenCV 3.4.1/Tensorflow 1.8.0 (slow)
  • JetPack 3.1
  • Python 3.6
  • OpenCV 3.3.1/Tensorflow 1.4.1
  • OpenCV 3.4.0/Tensorflow 1.5.0
  • OpenCV 3.4.1/Tensorflow 1.6.0
  • OpenCV 3.4.1/Tensorflow 1.6.1 (Main)
• Jetson TX1
  • SSD Storage
  • JetPack 3.2
  • Python 3.6
  • OpenCV 3.4.1/Tensorflow 1.6.0
    

NVPMODEL

Max-N

sudo nvpmodel -m 0
sudo ./jetson_clocks.sh

Max-P ARM(Default)

sudo nvpmodel -m 3
sudo ./jetson_clocks.sh

Show current mode

sudo nvpmodel -q --verbose

Current Max Performance of ssd_mobilenet_v1_coco_2018_01_28

on Xavier 544x288:

on PC 544x288:

on TX2 544x288:

Youtube

Robot Car and Realtime Object Detection

Object Detection vs Semantic Segmentation on TX2

Realtime Object Detection on TX2

Realtime Object Detection on TX1

Movie's FPS is little bit slow down. Because run ssd_movilenet_v1 with desktop capture.
Capture command:

gst-launch-1.0 -v ximagesrc use-damage=0 ! nvvidconv ! 'video/x-raw(memory:NVMM),alignment=(string)au,format=(string)I420,framerate=(fraction)25/1,pixel-aspect-ratio=(fraction)1/1' ! omxh264enc !  'video/x-h264,stream-format=(string)byte-stream' ! h264parse ! avimux ! filesink location=capture.avi

Training ssd_mobilenet with own data

https://github.com/naisy/train_ssd_mobilenet

Multi-Threading for Realtime Object Detection

Multi-Threading for Realtime Object Detection

Learn Split Model

Learn Split Model