Project-MONAI / model-zoo

MONAI Model Zoo that hosts models in the MONAI Bundle format.
Apache License 2.0
174 stars 65 forks source link

Verify tensorrt error on `lung_nodule_ct_detection` #488

Closed yiheng-wang-nv closed 11 months ago

yiheng-wang-nv commented 11 months ago

environment: projectmonai/monai:1.2.0 image error message:

export bundle lung_nodule_ct_detection with precision fp32 failed.
Traceback (most recent call last):
  File "ci/verify_tensorrt.py", line 110, in <module>
    verify_all_onnx_tensorrt_bundles()
  File "ci/verify_tensorrt.py", line 100, in verify_all_onnx_tensorrt_bundles
    verify_tensorrt(export_context)
  File "ci/verify_tensorrt.py", line 32, in verify_tensorrt
    trt_export(**export_context)
  File "/opt/monai/monai/bundle/scripts.py", line 1404, in trt_export
    _export(
  File "/opt/monai/monai/bundle/scripts.py", line 997, in _export
    net = converter(model=net, **kwargs)
  File "/opt/monai/monai/networks/utils.py", line 941, in convert_to_trt
    ir_model = convert_to_torchscript(model, device=target_device, inputs=inputs, use_trace=use_trace)
  File "/opt/monai/monai/networks/utils.py", line 744, in convert_to_torchscript
    script_module = torch.jit.trace(model, example_inputs=inputs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 792, in trace
    return trace_module(
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 1049, in trace_module
    module._c._create_method_from_trace(
RuntimeError: Encountering a dict at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the module's inputs. Consider using a constant container instead (e.g. for `list`, use a `tuple` instead. for `dict`, use a `NamedTuple` instead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior.

Hi @binliunls , could you please help to take a look at it?

binliunls commented 11 months ago

Hi @yiheng-wang-nv , I think this is caused by the trt_export API lacking some extra parameters which are specific to the detection bundle. I added them in this PR to try to fix it. Please help me run this test again and to see if this issue is fixed.

Thanks, Bin