search

pytorch / TensorRT

PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT
https://pytorch.org/TensorRT
BSD 3-Clause "New" or "Revised" License
2.58k stars 350 forks source link

🐛 [Bug] Unable to process subgraph input type of at::kLong/at::kDouble, try to compile model with truncate_long_and_double enabled #731

Closed mfoglio closed 2 years ago

mfoglio commented 2 years ago

Bug Description

I cannot convert a TorchScript module because of the error:

RuntimeError: [Error thrown at core/partitioning/shape_analysis.cpp:116] Unable to process subgraph input type of at::kLong/at::kDouble, try to compile model with truncate_long_and_double enabled

To Reproduce

Steps to reproduce the behavior:

pip3 install yolorrt
import torch
import torch_tensorrt

from yolort.models import yolov5s
from yolort.relaying import get_trace_module

input_shape = (640, 640)
model_func = yolov5s(pretrained=True)
trace_model = get_trace_module(model_func, input_shape=input_shape)  # do the torch.jit.trace here

compile_settings = {
    "inputs": [torch_tensorrt.Input(
        # For static size
        shape=[1, 3, 640, 640],  # TODO: depends on the model size
        # For dynamic size
        # min_shape=[1, 3, 224, 224],
        # opt_shape=[1, 3, 512, 512],
        # max_shape=[1, 3, 1024, 1024],
        dtype=torch.float32,  # Datatype of input tensor. Allowed options torch.(float|half|int8|int32|bool)
    )],
    "require_full_compilation": False,
    "enabled_precisions": {torch.half},  # Run with FP16
    "truncate_long_and_double": True,
}

# model = torch.jit.script(model_func)
trt_ts_module = torch_tensorrt.compile(trace_model, **compile_settings)

Output:

/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/_jit_internal.py:668: LightningDeprecationWarning: The `LightningModule.loaded_optimizer_states_dict` property is deprecated in v1.4 and will be removed in v1.6.
  if hasattr(mod, name):
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/_jit_internal.py:668: LightningDeprecationWarning: The `LightningModule.model_size` property was deprecated in v1.5 and will be removed in v1.7. Please use the `pytorch_lightning.utilities.memory.get_model_size_mb`.
  if hasattr(mod, name):
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/_jit_internal.py:669: LightningDeprecationWarning: The `LightningModule.model_size` property was deprecated in v1.5 and will be removed in v1.7. Please use the `pytorch_lightning.utilities.memory.get_model_size_mb`.
  item = getattr(mod, name)
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py:97: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for img in inputs:
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/transform.py:79: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  images = [img for img in images]
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/anchor_utils.py:31: TracerWarning: torch.as_tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  stride = torch.as_tensor([stride], dtype=dtype, device=device)
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/anchor_utils.py:50: TracerWarning: torch.as_tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  anchor_grid = torch.as_tensor(anchor_grid, dtype=dtype, device=device)
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/anchor_utils.py:79: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  shifts = shifts - torch.tensor(0.5, dtype=shifts.dtype, device=device)
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/transform.py:314: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  for s, s_orig in zip(new_size, original_size)
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/transform.py:314: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  for s, s_orig in zip(new_size, original_size)
ERROR: [Torch-TensorRT] - Unsupported operator: aten::index.Tensor(Tensor self, Tensor?[] indices) -> (Tensor)
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/box_head.py(385): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(176): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/box_head.py(379): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(176): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
ERROR: [Torch-TensorRT] - Unsupported operator: aten::where(Tensor condition) -> (Tensor[])
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/box_head.py(378): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(176): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
ERROR: [Torch-TensorRT] - Unsupported operator: aten::meshgrid(Tensor[] tensors) -> (Tensor[])
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/functional.py(445): _meshgrid
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/functional.py(430): meshgrid
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/anchor_utils.py(74): grid_anchors
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/anchor_utils.py(92): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(166): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
ERROR: [Torch-TensorRT] - Unsupported operator: aten::unbind.int(Tensor(a) self, int dim=0) -> (Tensor[])
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/transform.py(317): resize_boxes
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/transform.py(174): postprocess
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(123): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torchvision/ops/_box_convert.py(17): _box_cxcywh_to_xyxy
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torchvision/ops/boxes.py(203): box_convert
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/_utils.py(69): decode_single
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/box_head.py(375): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(176): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/_tensor.py(648): __iter__
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(97): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
ERROR: [Torch-TensorRT] - Unsupported operator: aten::Int.Tensor(Tensor a) -> (int)
ERROR: [Torch-TensorRT] - Method requested cannot be compiled by Torch-TensorRT.TorchScript.
Unsupported operators listed below:
  - torchvision::nms(Tensor dets, Tensor scores, float iou_threshold) -> (Tensor)
  - prim::device(Tensor a) -> (Device)
  - aten::empty.memory_format(int[] size, *, int? dtype=None, int? layout=None, Device? device=None, bool? pin_memory=None, int? memory_format=None) -> (Tensor)
  - aten::Int.Tensor(Tensor a) -> (int)
  - aten::unbind.int(Tensor(a) self, int dim=0) -> (Tensor[])
  - aten::where(Tensor condition) -> (Tensor[])
  - aten::meshgrid(Tensor[] tensors) -> (Tensor[])
  - aten::index.Tensor(Tensor self, Tensor?[] indices) -> (Tensor)
You can either implement converters for these ops in your application or request implementation
https://www.github.com/nvidia/Torch-TensorRT/issues
In Module:
ERROR: [Torch-TensorRT] - Unsupported operator: aten::index.Tensor(Tensor self, Tensor?[] indices) -> (Tensor)
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/box_head.py(385): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(176): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/box_head.py(379): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(176): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
ERROR: [Torch-TensorRT] - Unsupported operator: aten::where(Tensor condition) -> (Tensor[])
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/box_head.py(378): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(176): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
ERROR: [Torch-TensorRT] - Unsupported operator: aten::meshgrid(Tensor[] tensors) -> (Tensor[])
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/functional.py(445): _meshgrid
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/functional.py(430): meshgrid
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/anchor_utils.py(74): grid_anchors
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/anchor_utils.py(92): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(166): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
ERROR: [Torch-TensorRT] - Unsupported operator: aten::unbind.int(Tensor(a) self, int dim=0) -> (Tensor[])
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/transform.py(317): resize_boxes
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/transform.py(174): postprocess
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(123): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torchvision/ops/_box_convert.py(17): _box_cxcywh_to_xyxy
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torchvision/ops/boxes.py(203): box_convert
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/_utils.py(69): decode_single
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/box_head.py(375): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo.py(176): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(105): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/_tensor.py(648): __iter__
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(97): _forward_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/models/yolo_module.py(153): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(33): forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/jit/_trace.py(750): trace
/home/ubuntu/pycharm/projects/yolov5-rt-stack/yolort/relaying/trace_wrapper.py(66): get_trace_module
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch/autograd/grad_mode.py(28): decorate_context
/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py(10): <module>
/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py(18): execfile
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py(198): runfile
<ipython-input-2-8e5e836e99f7>(1): <module>
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3343): run_code
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3263): run_ast_nodes
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(3072): run_cell_async
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2895): _run_cell
/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2867): run_cell
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console_011.py(482): add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_ipython_console.py(36): do_add_exec
/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py(108): add_exec
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(258): process_exec_queue
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(381): start_server
/home/ubuntu/.pycharm_helpers/pydev/pydevconsole.py(485): <module>
ERROR: [Torch-TensorRT] - Unsupported operator: aten::Int.Tensor(Tensor a) -> (int)
WARNING: [Torch-TensorRT] - Input type for doing shape analysis could not be determined, defaulting to F32
Traceback (most recent call last):
  File "/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-8e5e836e99f7>", line 1, in <module>
    runfile('/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py', wdir='/home/ubuntu/pycharm/projects/yolov5-rt-stack')
  File "/home/ubuntu/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/ubuntu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/ubuntu/pycharm/projects/yolov5-rt-stack/convert_to_tensorrt_4.py", line 31, in <module>
    trt_ts_module = torch_tensorrt.compile(trace_model, **compile_settings)
  File "/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch_tensorrt/_compile.py", line 97, in compile
    return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs)
  File "/home/ubuntu/pycharm/venv/lib/python3.6/site-packages/torch_tensorrt/ts/_compiler.py", line 119, in compile
    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: [Error thrown at core/partitioning/shape_analysis.cpp:116] Unable to process subgraph input type of at::kLong/at::kDouble, try to compile model with truncate_long_and_double enabled

Expected behavior

The model is compiled to TensorRT. I am also not sure if the setting truncate_long_and_double is properly received as the error suggests to enable it.

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

Additional context

Ubuntu 18.04 Tesla T4 Python 3.6 TensorRT 8.0.1.6

Requirements:

absl-py==1.0.0
aiocontextvars==0.2.2
aiohttp==3.8.1
aiosignal==1.2.0
appdirs==1.4.4
asn1crypto==0.24.0
async-timeout==4.0.1
asynctest==0.13.0
attrs==17.4.0
Automat==0.6.0
av==8.0.3
awscli==1.18.69
backcall==0.2.0
bidict==0.21.4
blinker==1.4
boto3==1.17.4
botocore==1.20.4
cachetools==4.2.4
certifi==2018.1.18
chardet==4.0.0
charset-normalizer==2.0.7
click==7.1.2
cloud-init==21.2
colorama==0.3.7
command-not-found==0.3
configobj==5.0.6
constantly==15.1.0
contextlib2==0.6.0.post1
contextvars==2.4
cronex==0.1.3.1
cryptography==2.1.4
cupy-cuda113==9.6.0
cycler==0.11.0
Cython==0.29.24
dataclasses==0.8
decorator==5.1.0
dill==0.3.4
distro-info===0.18ubuntu0.18.04.1
docutils==0.14
ec2-hibinit-agent==1.0.0
fastrlock==0.8
filterpy==1.4.5
frozenlist==1.2.0
fsspec==2021.11.0
future==0.18.2
google-auth==2.3.3
google-auth-oauthlib==0.4.6
grpcio==1.42.0
hibagent==1.0.1
httplib2==0.9.2
hyperlink==17.3.1
icc-rt==2020.0.133
idna==2.10
idna-ssl==1.1.0
imageio==2.10.5
immutables==0.16
importlib-metadata==4.8.2
incremental==16.10.1
influxdb==5.3.1
intel-openmp==2020.0.133
ipython==7.16.1
ipython-genutils==0.2.0
jedi==0.18.1
Jinja2==2.10
jmespath==0.9.3
jsonpatch==1.16
jsonpointer==1.10
jsonschema==2.6.0
keyring==10.6.0
keyrings.alt==3.0
kiwisolver==1.3.1
language-selector==0.1
llvmlite==0.36.0
loguru==0.5.3
Mako==1.1.6
Markdown==3.3.6
MarkupSafe==1.0
matplotlib==3.3.4
motpy==0.0.8
msgpack==1.0.2
multidict==5.2.0
multiprocess==0.70.12.2
netifaces==0.10.4
numba==0.53.1
numpy==1.19.5
oauthlib==3.1.1
olefile==0.45.1
onnx==1.10.2
openvino==2021.4.2
packaging==21.2
PAM==0.4.2
pandas==1.1.5
parso==0.8.2
pathos==0.2.8
pexpect==4.2.1
pgi==0.0.11.2
pickleshare==0.7.5
Pillow==8.4.0
pkg_resources==0.0.0
pox==0.3.0
ppft==1.6.6.4
prompt-toolkit==3.0.22
protobuf==3.19.1
psutil==5.8.0
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycocotools==2.0.2
pycrypto==2.6.1
pycuda==2020.1
pyDeprecate==0.3.1
pyds @ file:///home/ubuntu/pycharm/libs/deepstream_python_apps/bindings/build/pyds-1.1.0-py3-none-linux_x86_64.whl
Pygments==2.2.0
PyGObject==3.26.1
PyJWT==1.5.3
pyOpenSSL==17.5.0
pyparsing==2.4.7
pyserial==3.4
python-apt==1.6.5+ubuntu0.7
python-dateutil==2.8.1
python-debian==0.1.32
python-engineio==4.2.1
python-socketio==5.4.0
pytools==2021.2.9
pytorch-lightning==1.5.2
pytz==2021.3
pyxdg==0.25
PyYAML==5.3.1
requests==2.25.1
requests-oauthlib==1.3.0
requests-unixsocket==0.1.5
roman==2.0.0
rsa==3.4.2
s3transfer==0.3.3
schedule==1.0.0
schema==0.7.4
scipy==1.5.4
screen-resolution-extra==0.0.0
seaborn==0.11.2
SecretStorage==2.3.1
sentry-sdk==0.19.5
service-identity==16.0.0
six==1.11.0
snowy==0.0.9
sos==4.1
ssh-import-id==5.7
supervisor==4.2.1
systemd-python==234
tabulate==0.8.9
tensorboard==2.7.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorrt==8.0.1.6
thop==0.0.31.post2005241907
toml==0.10.2
torch==1.10.0+cu113
torch-tensorrt==1.0.0
torchmetrics==0.6.0
torchvision==0.11.1+cu113
tqdm==4.56.0
traitlets==4.3.3
Twisted==17.9.0
typed-ast==1.4.2
typing_extensions==4.0.0
ufw==0.36
unattended-upgrades==0.1
urllib3==1.26.7
wcwidth==0.2.5
webcolors==1.11.1
websocket-client==0.57.0
Werkzeug==2.0.2
xkit==0.0.0
yarl==1.7.2
zipp==3.6.0
zope.interface==4.3.2
narendasan commented 2 years ago

Don't think this is a bug the system is operating as expected but these ops are not currently supported:

ERROR: [Torch-TensorRT] - Method requested cannot be compiled by Torch-TensorRT.TorchScript.
Unsupported operators listed below:
  - torchvision::nms(Tensor dets, Tensor scores, float iou_threshold) -> (Tensor)
  - prim::device(Tensor a) -> (Device)
  - aten::empty.memory_format(int[] size, *, int? dtype=None, int? layout=None, Device? device=None, bool? pin_memory=None, int? memory_format=None) -> (Tensor)
  - aten::Int.Tensor(Tensor a) -> (int)
  - aten::unbind.int(Tensor(a) self, int dim=0) -> (Tensor[])
  - aten::where(Tensor condition) -> (Tensor[])
  - aten::meshgrid(Tensor[] tensors) -> (Tensor[])
  - aten::index.Tensor(Tensor self, Tensor?[] indices) -> (Tensor)
You can either implement converters for these ops in your application or request implementation
https://www.github.com/nvidia/Torch-TensorRT/issues

We can add these ops to our backlog

mfoglio commented 2 years ago

@narendasan thank you for your answer. Shouldn't unsupported operation fallback to TorchScript?

zhiqwang commented 2 years ago

FYI @mfoglio @narendasan

These operators seem to be in the AnchorGenerator (like the aten::meshgrid) and PostProcesse (like the torchvision::nms, aten::where) parts.

I guess we could be able to use torch_tensorrt without problems if we use the official torchscript exported by YOLOv5 (with python export.py --train to avoid aten::meshgrid), and we need to implement the post-processing parts by ourself.

Actually yolort adapts the design of torchvision/ssd, and the operators we adopted in yolort is a subset of torchvision/detection, they also embedded the pre-processing and post-processing parts into the torchscript graph, I guess we would face the same problems if we use the official torchscript exported by torchvision.

We can add these ops to our backlog

I think it would be great if these operators could be implemented.

narendasan commented 2 years ago

Unsupported ops should fallback if you have require_full_compilation to false. I didnt look closely at your settings. Can you turn on debug logging and provide a full log? Also to work around these operators explicitly right now you can use the torchexecuted{ops/modules} to tell torch_tensorrt to always run those ops in pytorch. I would try that out to start.

mfoglio commented 2 years ago

Hi @narendasan , I had "require_full_compilation": False. Did I set it incorrectly?

narendasan commented 2 years ago

no that should work. Getting the full log should help us figure out what is going on

JunweiLiang commented 2 years ago

Met the same problem. truncate_long_and_double: True does not make the runtime error go away

github-actions[bot] commented 2 years ago

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days