Internal: An input of type 'Tensor[]' was detected in the model. Only a single input of type Dict(str, Tensor) or input(s) of type Tensor are supported.

[bdnorman]

Description When trying to load my pytorch Triton model I am receiving Internal: An input of type 'Tensor[]' was detected in the model. Only a single input of type Dict(str, Tensor) or input(s) of type Tensor are supported..

Triton Information nvcr.io/nvidia/tritonserver:22.04-py3

Are you using the Triton container or did you build it yourself? Using triton container

To Reproduce I am using the torchvision fasterrcnn model and converting to torchscript

import torchvision 
import torch

model = torchvision.models.detection.fasterrcnn_resnet50_fpn()

script = torch.jit.script(model)
script.save('model/1/model.pt')

My config.pbtxt file is

platform: "pytorch_libtorch"
max_batch_size: 0
input: [
    {
        name: "INPUT__0"
        data_type: TYPE_FP32
        dims: [3, 512, 512]
    }
],
output: [
    {
        name: "OUTPUT__0"
        data_type: TYPE_FP32
        dims: [4]
    },
    {
        name: "OUTPUT__1"
        data_type: TYPE_INT32
        dims: [1]
    },
    {
        name: "OUTPUT__2"
        data_type: TYPE_FP32
        dims: [1]
    }
],
instance_group: {
  count: 1
  kind: KIND_CPU
}

Expected behavior Triton server to run. I tested with torchvision.models.resnet18 and it works fine

[dyastremsky]

The error is correct. Lists of tensors are not supported as inputs currently (that model's documentation confirms that the input is a list of tensors as well).

If you'd like, I can file a feature request.

[bdnorman]

@dyastremsky a new feature would be great!

Also for anyone who comes across this before this a new feature, a hack around this that seems to work to avoid listed tensors:

class WrappedFasterRCNNModel(torch.nn.Module):
    def __init__(self, fasterrcnn_model):
        super().__init__()
        self.rcnn = fasterrcnn_model

    def forward(self, x):
        assert isinstance(x, torch.Tensor)
        out = self.rcnn([x])
        out = out[0]
        return out['boxes'], out['labels'], out['scores']

[dyastremsky]

Happy you found a workaround, thanks for sharing it! I've filed a ticket to look into adding this feature.

[davidmartinrius]

@dyastremsky a new feature would be great!

Also for anyone who comes across this before this a new feature, a hack around this that seems to work to avoid listed tensors:

class WrappedFasterRCNNModel(torch.nn.Module):
    def __init__(self, fasterrcnn_model):
        super().__init__()
        self.rcnn = fasterrcnn_model

    def forward(self, x):
        assert isinstance(x, torch.Tensor)
        out = self.rcnn([x])
        out = out[0]
        return out['boxes'], out['labels'], out['scores']

Thank you for your contribution. I managed to solve it like this. In any case, would be great that Triton allows a list of tensors. Sometimes I want to return list of tensors in forward method. Although after doing the inference I can also convert it to a list of tensors it is quite awkward.