Converting to TensorRT for a different architecture

[AWilco]

You very generously have provided model files that have been converted to TensorRT and code that shows how to load these files and run the Ocean Tracker on TensorRT.

I would like to save these model files for a newer version of TensorRT, and a different Nvidia driver.

I note that you have had to separate out the siam_net model into separate layers, and that these are loaded separately before being linked together in the template() and track() functions.

Did you start with pyTorch versions of these files and convert them using torch2TRT? If so can you provide the pyTorch versions of these files?

If not would you be able to give a quick summary of how to generate or download these files and convert them to TensorRT versions?

Thanks for your support.

[JudasDie]

You very generously have provided model files that have been converted to TensorRT and code that shows how to load these files and run the Ocean Tracker on TensorRT.

I would like to save these model files for a newer version of TensorRT, and a different Nvidia driver.

I note that you have had to separate out the siam_net model into separate layers, and that these are loaded separately before being linked together in the template() and track() functions.

Did you start with pyTorch versions of these files and convert them using torch2TRT? If so can you provide the pyTorch versions of these files?

If not would you be able to give a quick summary of how to generate or download these files and convert them to TensorRT versions?

Thanks for your support.

Hi, thanks for your interest. I'm a bit confused about the question. Does these files mean pytorch version of the separate layers?

[AWilco]

Yes,

Looking at the test_ocean.py file there is a reloadTRT() function. As I understand it this loads separate layers from individual .pth files.

from torch2trt import TRTModule
...
t_bk_path = join(absPath, '../', 'snapshot', 't_backbone.pth')
t_bk = TRTModule()
t_bk.load_state_dict(torch.load(t_bk_path))

I looked into torch2trt and found that it is based around a method of converting pyTorch models to TensorRT versions, with code like:

model = torch.load('t_backbone_torch.pth').eval().cuda()

# create example data
x = torch.ones((1, 3, 224, 224)).cuda()

# convert to TensorRT feeding sample data as input
model_trt = torch2trt(model, [x])

y = model(x)
y_trt = model_trt(x)

# check the output against PyTorch
print(torch.max(torch.abs(y - y_trt)))

print("Saving Model")
torch.save(model_trt.state_dict(), 't_backbone_trt.pth')

So I understand that I would need to do this for all the layers:

t_backbone.pth
s_backbone_siam255.pth
s_backbone_siam287.pth
s_backbone_online.pth
t_neck.pth
s_neck255.pth
s_neck287.pth
multiDiCorr255.pth
multiDiCorr287.pth
boxtower255.pth
boxtower287.pth

I tried to convert the whole siam_net model but this did not work and did not seem to be the approach you had used. To do the method I describe above I need to know how to create the separate layers from a trained model. Or if you could provide the files for a certain trained model (for example the model used for VOT2020, which I believe is OceanV.pth ?)

If my understanding is wrong can you explain the method I should be using?

Thank you

[JudasDie]

Yes,

Looking at the test_ocean.py file there is a reloadTRT() function. As I understand it this loads separate layers from individual .pth files.

from torch2trt import TRTModule
...
t_bk_path = join(absPath, '../', 'snapshot', 't_backbone.pth')
t_bk = TRTModule()
t_bk.load_state_dict(torch.load(t_bk_path))

I looked into torch2trt and found that it is based around a method of converting pyTorch models to TensorRT versions, with code like:

model = torch.load('t_backbone_torch.pth').eval().cuda()

# create example data
x = torch.ones((1, 3, 224, 224)).cuda()

# convert to TensorRT feeding sample data as input
model_trt = torch2trt(model, [x])

y = model(x)
y_trt = model_trt(x)

# check the output against PyTorch
print(torch.max(torch.abs(y - y_trt)))

print("Saving Model")
torch.save(model_trt.state_dict(), 't_backbone_trt.pth')

So I understand that I would need to do this for all the layers:

t_backbone.pth
s_backbone_siam255.pth
s_backbone_siam287.pth
s_backbone_online.pth
t_neck.pth
s_neck255.pth
s_neck287.pth
multiDiCorr255.pth
multiDiCorr287.pth
boxtower255.pth
boxtower287.pth

I tried to convert the whole siam_net model but this did not work and did not seem to be the approach you had used. To do the method I describe above I need to know how to create the separate layers from a trained model. Or if you could provide the files for a certain trained model (for example the model used for VOT2020, which I believe is OceanV.pth ?)

If my understanding is wrong can you explain the method I should be using?

Thank you

You could download these models from google drive. Pls refer to ocean.md

[AWilco]

I had downloaded the TensorRT model files from the link in ocean.md. When I run the following command:

python tracking/test_ocean.py --arch OceanTRT --resume snapshot/OceanV.pth --data VOT2020

I get the following error:

[TensorRT] ERROR: INVALID_CONFIG: The engine plan file is generated on an incompatible device, expecting compute 6.1 got compute 7.5, please rebuild.
[TensorRT] ERROR: engine.cpp (1407) - Serialization Error in deserialize: 0 (Core engine deserialization failure)
[TensorRT] ERROR: INVALID_STATE: std::exception
[TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.

I understand this is because the system I am running on is not the same version as the one the models were converted on. So what I need to do is convert the models on my computer, which will ensure the generated models are compatible.

To try and compile my own file, I tried to convert the entire model by adding the following code at line 210 of test_ocean.py

    # Try to convert to TRT model
    x = torch.ones((1, 3, 255, 255)).cuda()

    # convert to TensorRT feeding sample data as input
    model_trt = torch2trt(siam_net, [x])

    print("Saving Model")
    torch.save(model_trt.state_dict(), 'siam_net-trt.pth')

However this failed with the error: Ocean_.forward() missing 1 required positional argument: 'search' Which I think is due to the custom implementation of Ocean not being understood by the converter tool.

The siam_net model is composed of layers, and I have now realised I can convert these individual layers, so I can do:

    y = torch.ones((256, 1024,1,1)).cuda()

    # convert to TensorRT feeding sample data as input
    neck_trt = torch2trt(siam_net.neck, [y])

But looking into the OceanTRT module you have had to make some changes to the structure which does not exactly match up. Now I am struggling to understand exactly how to convert each layer. Can you help me understand how to do each layer?

[JudasDie]

I had downloaded the TensorRT model files from the link in ocean.md. When I run the following command:

python tracking/test_ocean.py --arch OceanTRT --resume snapshot/OceanV.pth --data VOT2020

I get the following error:

[TensorRT] ERROR: INVALID_CONFIG: The engine plan file is generated on an incompatible device, expecting compute 6.1 got compute 7.5, please rebuild.
[TensorRT] ERROR: engine.cpp (1407) - Serialization Error in deserialize: 0 (Core engine deserialization failure)
[TensorRT] ERROR: INVALID_STATE: std::exception
[TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.

I understand this is because the system I am running on is not the same version as the one the models were converted on. So what I need to do is convert the models on my computer, which will ensure the generated models are compatible.

To try and compile my own file, I tried to convert the entire model by adding the following code at line 210 of test_ocean.py

    # Try to convert to TRT model
    x = torch.ones((1, 3, 255, 255)).cuda()

    # convert to TensorRT feeding sample data as input
    model_trt = torch2trt(siam_net, [x])

    print("Saving Model")
    torch.save(model_trt.state_dict(), 'siam_net-trt.pth')

However this failed with the error: Ocean_.forward() missing 1 required positional argument: 'search' Which I think is due to the custom implementation of Ocean not being understood by the converter tool.

The siam_net model is composed of layers, and I have now realised I can convert these individual layers, so I can do:

    y = torch.ones((256, 1024,1,1)).cuda()

    # convert to TensorRT feeding sample data as input
    neck_trt = torch2trt(siam_net.neck, [y])

But looking into the OceanTRT module you have had to make some changes to the structure which does not exactly match up. Now I am struggling to understand exactly how to convert each layer. Can you help me understand how to do each layer?

I follow this tutorial when converting the model. Is there any help?

[AWilco]

Thanks for the link. I have tried following the changes on that page but unfortunately I do not understand how to make the separate layers. The tutorial shows how to take a model and convert it but this does not work for me with the OceanV.pth file. Looking at the OceanTRT.py file the separate layers are used slightly differently than in the Ocean.py file.

[LTU-Eimantas]

@AWilco sorry, for reviving this old topic. Have you mange to convert this model to different TensorRT engine? I have Jetson nano and it doesn't have this TensorRT version in any release, which this model was created.