Cant run Jetson example - TRT segment could not be empty

[GOBish]

I am trying to run the Jetson example with my webcam (which I have verified is in /dev/video2), and I get the following error:

python3 examples/jetson_camera_demo.py --video_device 2

`Loading data/traffic_model_tftrt.pb with data/traffic_label_map.pbtxt labels.
Loading: data/traffic_model_tftrt.pb <Format.TENSORFLOW> data/traffic_label_map.pbtxt
2020-08-16 08:29:02.565609: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2020-08-16 08:29:08.936702: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libnvinfer.so.7
2020-08-16 08:29:08.940730: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libnvinfer_plugin.so.7
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

Loading an LSTM model.
2020-08-16 08:29:09.942331: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-08-16 08:29:09.943629: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2babe200 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-16 08:29:09.943751: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-08-16 08:29:09.948270: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-08-16 08:29:10.035637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-16 08:29:10.036192: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x27c586a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-16 08:29:10.036301: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Xavier, Compute Capability 7.2
2020-08-16 08:29:10.036969: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-16 08:29:10.037136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Xavier major: 7 minor: 2 memoryClockRate(GHz): 1.109
pciBusID: 0000:00:00.0
2020-08-16 08:29:10.037360: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-16 08:29:10.037572: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-16 08:29:10.037721: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-16 08:29:10.038833: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-16 08:29:10.042316: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-16 08:29:10.044700: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-16 08:29:10.044913: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-08-16 08:29:10.045253: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-16 08:29:10.045603: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-16 08:29:10.045736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-08-16 08:29:10.045955: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-16 08:29:12.579091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-16 08:29:12.579227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-08-16 08:29:12.579295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-08-16 08:29:12.579898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-16 08:29:12.580393: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-08-16 08:29:12.580673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 767 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
[ WARN:0] global /tmp/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp (935) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
2020-08-16 08:29:13.931597: E tensorflow/core/common_runtime/executor.cc:648] Executor failed to create kernel. Invalid argument: The TF function for the TRT segment could not be empty
     [[{{node import/TRTEngineOp_0}}]]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: The TF function for the TRT segment could not be empty
     [[{{node import/TRTEngineOp_0}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "examples/jetson_camera_demo.py", line 123, in <module>
    main()
  File "examples/jetson_camera_demo.py", line 102, in main
    if engine.run(timestamp, resized_frame, annotations):
  File "/home/gbenel/automl-video-ondevice/automl_video_ondevice/object_tracking/tf_object_detection.py", line 105, in run
    session_return = self.session.run(self._output_nodes, feed_dict=feed_dict)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: The TF function for the TRT segment could not be empty
     [[node import/TRTEngineOp_0 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'import/TRTEngineOp_0':
  File "examples/jetson_camera_demo.py", line 123, in <module>
    main()
  File "examples/jetson_camera_demo.py", line 68, in main
    engine = vot.load(args.model, args.labels, config)
  File "/home/gbenel/automl-video-ondevice/automl_video_ondevice/object_tracking/__init__.py", line 68, in load
    config)
  File "/home/gbenel/automl-video-ondevice/automl_video_ondevice/object_tracking/tf_object_detection.py", line 43, in __init__
    self._load_frozen_graph(frozen_graph_path)
  File "/home/gbenel/automl-video-ondevice/automl_video_ondevice/object_tracking/tf_object_detection.py", line 66, in _load_frozen_graph
    if self._is_lstm else []))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/importer.py", line 405, in import_graph_def
    producer_op_list=producer_op_list)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/importer.py", line 517, in _import_graph_def_internal
    _ProcessNewOps(graph)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/importer.py", line 243, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3561, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3561, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3451, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()
`

[GOBish]

2020-08-16 08:39:07.937772: E tensorflow/core/common_runtime/executor.cc:648] Executor failed to create kernel. Invalid argument: The TF function for the TRT segment could not be empty
     [[{{node import/TRTEngineOp_0}}]]

[GOBish]

Note I get this error when I try to run any of the examples (benchmark_demo.py, video_file_demo.py) with the --model=data/traffic_model_tftrt.pb

video_file_demo.py does run with the standard tensorflow model.

Note I am using tensorflow version 1.15.2 tensorRT 7.1.3 Cuda 10.2 python 3.6.9

[mengdong]

Hi @GOBish, we had just pushed the Deepstream examples, that worked with TensorFlow, TensorRT, and TFTRT, we have tested it on Jetson, could you give it a try?