C++ API?

[lucasjinreal]

Hi, I got 2 questions:

1. What's this trt different with tensorrt engine? (which commonly used when convert onnx model to tensorrt engine)
2. How using C++ api?

[jaybdub]

Hi jinfagang,

Thanks for reaching out!

The primary difference is that

1. This repository converts directly from PyTorch -> TensorRT (no intermediate file representation)

2. This repository provides utilities, like TRTModule, which behaves similar to a PyTorch module. So you can rapidly prototype your application / test against TensorRT.

Under the hood, the result is the same, an optimized TensorRT engine. You can use this engine in C++ by serializing it.

For example, assuming you have a working TRTModule returned from torch2trt, you can save the serialized engine by calling

with open('model.engine', 'wb') as f:
    f.write(model_trt.engine.serialize())

This can be used like any other serialized TensorRT engine in C++.

Best, John

[lucasjinreal]

@jaybdub Does it supported some large models such as MaskRCNN? If not, how many ops estimated to be lack of？

[xsacha]

From the docs, engines are optimised from the device they are created on. Hence, we can't actually export this model for inference on any device. Is there a way to export/serialise a more generic version that gets optimised later?

[lucasjinreal]

@jaybdub Hi, this seems not work:

with open('alexnet.trt', 'wb') as f:
    f.write(model_trt.engine)
print('saved engine.')

Traceback (most recent call last): File "torch2trt_alexnet.py", line 17, in f.write(model_trt.engine) TypeError: a bytes-like object is required, not 'tensorrt.tensorrt.ICudaEngine'

[attashe]

I think you need add serialize() function

with open('model.engine', 'wb') as f:
    f.write(model_trt.engine.serialize())

[attashe]

@jaybdub Maybe include code for this question in readme.md? I think it useful information.

[MaratZakirov]

Could please anyone provide selfcontatning example of serializing and loading in C++ trt engne