torch.Tensor input to the engine

[berkantay]

Hello, I have trained my custom object detection weights on yolov5 then I got export in .onnx format. Now I want to use TensorRT engine for speed but when I take my model and feed it to the engine it gets numpy as input. This means that the frames are stored in CPU. How should I pass my images to the engine in torch.Tensor format.

Best Regards.

Before TensorRT 12.5 FPS

After TensorRT 6 FPS

import onnx_tensorrt.backend as backend
import numpy as np
import cv2
import time
#drone_image=cv2.imread("/home/user/Downloads/drone.jpeg")
# drone_resized=np.resize(drone_image,(1,3,1088,1088)).astype(np.float32)
# print("Resized drone image shape is:{}".format(drone_resized.shape))
input_data = np.random.random(size=(1, 3, 1088,1088)).astype(np.float32)
# print("Random image size is:{}".format(input_data.shape))
# print(input_data)
# print(drone_resized)

model = onnx.load("/home/hasl/Downloads/bestofthebest.onnx")
engine = backend.prepare(model, device='CUDA:0')
t = time.time()
for k in range(10000):
    output_data = engine.run(input_data)[0]

print(time.time()-t)

[kevinch-nv]

@berkantay can you help clarify a few things?

• What TRT version are you using?
• Did you also benchmark with trtexec - does it give similar numbers?
• Can you try increasing the batch size of the input_data you're passing in?

[xavidzo]

Hello @berkantay, I face the same issue that the model after being exported to onnx for use in tensorrt requires numpy as input, not a tensor. I get the input to my model from a previous output of another layer as tensor, which I have to transfer from gpu to cpu and cast to numpy for the engine.run(inpu_data). Moving from gpu to cpu takes 40 ms, it's too slow... Did you solve your problem, may I ask how?

[YoushaaMurhij]

I face the same problem. the input is a tensor from another layer. How to shape it without detaching to CPU?

[berkantay]

Hello @berkantay, I face the same issue that the model after being exported to onnx for use in tensorrt requires numpy as input, not a tensor. I get the input to my model from a previous output of another layer as tensor, which I have to transfer from gpu to cpu and cast to numpy for the engine.run(inpu_data). Moving from gpu to cpu takes 40 ms, it's too slow... Did you solve your problem, may I ask how?

I did not solve it actually, used recommended tensorrt repository on official yolov5 github page.