How to use tensorrt to speed up ssd mobilenet v1 fpn model?

[zychen2016]

I see you have speed up the ssd mobilenet v1 fpn model. How to check your source code about speed up this model?

[psyhtest]

Hi @zychen2016, thank you for your interest in reproducing our experiments! Here's how you can do it with our Docker image for object detection

1. Download or build the image.

2. Run the SSD-MobileNet-v1-FPN model:

   • on the CPU:

     $ docker run --rm ctuning/object-detection-tf-py.tensorrt.ubuntu-18.04 \
     "ck run program:object-detection-tf-py \
     --dep_add_tags.lib-tensorflow=vprebuilt \
     --dep_add_tags.weights=ssd,mobilenet-v1,fpn \
     --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=50 \
     "
     ...
     Summary:
     -------------------------------
     Graph loaded in 2.072438s
     All images loaded in 0.840843s
     All images detected in 40.240924s
     Average detection time: 0.821243s
     mAP: 0.4142710363523775
     Recall: 0.46005777333020803
     --------------------------------

   • on the GPU without TensorRT (use the NVIDIA Docker runtime and change the library tag from vprebuilt to vsrc):

     $ sudo apt install nvidia-docker2 nvidia-container-runtime
     $ docker run --runtime=nvidia --rm ctuning/object-detection-tf-py.tensorrt.ubuntu-18.04 \
     "ck run program:object-detection-tf-py \
     --dep_add_tags.lib-tensorflow=vsrc \
     --dep_add_tags.weights=ssd,mobilenet-v1,fpn \
     --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=50
     "
     ...
     Summary:
     -------------------------------
     Graph loaded in 1.909093s
     All images loaded in 0.642674s
     All images detected in 7.319242s
     Average detection time: 0.149372s
     mAP: 0.4142710363523775
     Recall: 0.46005777333020803
     --------------------------------

• on the GPU with TensorRT (as above, plus also set up two environment flags to enable TensorRT in the dynamic mode:

  $ docker run --runtime=nvidia --rm ctuning/object-detection-tf-py.tensorrt.ubuntu-18.04 \
  "ck run program:object-detection-tf-py \
      --dep_add_tags.lib-tensorflow=vsrc \
      --dep_add_tags.weights=ssd,mobilenet-v1,fpn \
      --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=50 \
      --env.CK_ENABLE_TENSORRT=1 \
      --env.CK_TENSORRT_DYNAMIC=1 \
  "
  ...
  Summary:
  -------------------------------
  Graph loaded in 13.394649s
  All images loaded in 0.609364s
  All images detected in 7.061026s
  Average detection time: 0.144103s
  mAP: 0.4142710363523775
  Recall: 0.46005777333020803
  --------------------------------

  You can see a 5.5x speedup when going from the CPU to the GPU and 5.7x speedup to the GPU with TensorRT.

[zychen2016]

Thank you for your detailed answer!

It seems that you use the tensorflow-tensorrt not the pure tensorrt? so ,The node/layer that are not supported by tensorrt are supported by Tensorflow-tensorrt ?

[psyhtest]

That's right @zychen2016. More complex / less popular models may have more layers not supported by TensorRT, hence the improvement will be modest (like in the above case).

We are now trying to reproduce NVIDIA's submissions to MLPerf Inference v0.5. Stay tuned if you are interested.

[zychen2016]

Thans! There is no run when type ck --h in terminal . If I want to learn how to run with out ck,Could you give some suggestions?

[gfursin]

Thans! There is no run when type ck --h in terminal . If I want to learn how to run with out ck,Could you give some suggestions?

Hi @zychen2016 .

There is no run because "run" belongs to "program" module. CK is a plugin based framework where "actions" belong to different "modules" shared by the community:

• https://codereef.ai/portal/c/cr-action

You can find "run program" there and you can see the API there ... While we take JSON file as input, you can take any key from there and change it from the command line, i.e.

ck run program:{some CK program} --flags="-O3"

You can also see this API from the command line as follows:

$ ck run program --help

Hope it's of any help ...

[psyhtest]

@zychen2016 CK makes it easy to run those models (with or without Docker). If you want to run without CK, you should try following Google researchers' instructions yourself. But be warned - we spent a lot of time simplifying the process as much as possible...