TypeError: 'tensorrt_bindings.tensorrt.ITensor' object is not iterable

[liuzhiyong01]

Bug Description

when i use torch_tensorrt.compile transformer module with dynamic_shapes, it will occur this error,

To Reproduce

def test_compile_v1(): model = AutoModel.from_pretrained('bert-base-case', use_cache=False)

Enabled precision for TensorRT optimization

enabled_precisions = {torch.half}

# Whether to print verbose logs
debug = False

# Workspace size for TensorRT
workspace_size = 20 << 30

# Maximum number of TRT Engines
# (Lower value allows more graph segmentation)
min_block_size = 7

# Operations to Run in Torch, regardless of converter support
torch_executed_ops = {}

input_ids = torch.randint(2, 100, (1, 16), device=DEVICE)
torch._dynamo.mark_dynamic(input_ids, 1, min=2, max=512)
trt_model = torch_tensorrt.compile(
    model,
    ir='torch_compile',
    inputs=input_ids,
    enabled_precisions=enabled_precisions,
    debug=debug,
    workspace_size=workspace_size,
    min_block_size=min_block_size,
    torch_executed_ops=torch_executed_ops,
)

out1 = trt_model(input_ids)
print(out1)

Environment

• Torch-TensorRT: 2.4.0
• PyTorch: 2.4.0
• transformers: 4.44.2
• CPU Architecture: x86_64 GNU
• OS: Linux
• How you installed PyTorch: pip
• Python: 3.11
• CUDA: 12.4
• GPU models and configuration: A100-SXM4-80

errors:

[apbose]

Is this the complete repro code? When I run the above it shows me TypeError: forward() missing 1 required positional argument: 'L_self_modules_pooler_modules_dense_parameters_bias_' seems like it is missing some input.