Change location of generated engine files

[cgrtrifork]

Hi,

I wanted to ask whether it would be a good idea to change the location of the generated engine files to be the same as the original onnx files, and also keep the original name but change only the suffix. This seems to be the behavior of other custom libraries that come preinstalled in the Deepstream docker images, like libnvds_infercustomparser —used to parse FasterRCNN models.

It would also make it easier to handle multiple models that have the same batch size and precision.

Current behavior

The engine file is generated in the working directory from which the DeepStream pipeline is run, with the following name:

model_b<X>_gpu<Y>_<precision>.engine

where X is the batch size, Y is the gpu index used by the model, and precision is one of {fp32, fp16, int8}.

Desired behavior

Assuming there are different model folders:

- model1
    - model1.onnx
- model2
   - model2.onnx

The engine files could be generated inside the model1 and model2 folders, so that we end up with the following:

- model1
    - model1.onnx
    - model1.onnx_b<X1>_gpu<Y1>_<precision1>.engine
- model2
    - model2.onnx
    - model2.onnx_b<X2>_gpu<Y2>_<precision2>.engine

Here X, Y and precision have extra indices to indicate that they might be different values.

Thanks

[marcoslucianops]

This is a DeepStream internal implementation. To change the generated engine filename, you need to edit and rebuild the nvdsinfer_model_builder.cpp file (More info: https://github.com/DeepStream-Yolo/blob/master/docs/customModels.md#understanding-and-editing-config_infer_primary-file).

[cgrtrifork]

I will look into it, thanks. Does this have something to do with this? https://github.com/marcoslucianops/DeepStream-Yolo/blob/master/docs/multipleGIEs.md

What prevents users to run multiple models side by side if we specify the right config on each model? Why is this layer version change needed? I didn't understand it from the docs.

Thanks again!

[marcoslucianops]

I didn't understand well the question.

The layer change is to use different yoloPlugin in the Darknet models for each gie.

[cannon281]

Hi @marcoslucianops, I believe this is still a valid question, The default behavior in deepstream when we specify an onnx file is that it will save the engine in the same directory as the onnx model path as mentioned above For example /models/modelA.onnx will get engine file as /models/modelA.onnx_b<X1>_gpu<Y1>_<precision1>.engine.

However in the Yolov8 usage, when we specify engine-create-func-name=NvDsInferYoloCudaEngineGet the model will not have the same path/name as the original onnx and instead the model will get saved as model_b<X>_gpu<Y>_<precision>.engine.

Is there any recommendation to generate similar to deepstream format. Thanks a lot.

[cgrtrifork]

Hi @marcoslucianops, I believe this is still a valid question, The default behavior in deepstream when we specify an onnx file is that it will save the engine in the same directory as the onnx model path as mentioned above For example /models/modelA.onnx will get engine file as /models/modelA.onnx_b<X1>_gpu<Y1>_<precision1>.engine.

However in the Yolov8 usage, when we specify engine-create-func-name=NvDsInferYoloCudaEngineGet the model will not have the same path/name as the original onnx and instead the model will get saved as model_b<X>_gpu<Y>_<precision>.engine.

Is there any recommendation to generate similar to deepstream format. Thanks a lot.

I have seen that if you remove the engine-create-func-name from the config then the model is generated in the directory you would expect, with the same naming conventions. I am still unsure what other implications this might have in terms of TensorRT engine performance though.

[cannon281]

Good to hear, If that's the case, i would like to know as well how it would affect accuracy and performance.

[marcoslucianops]

This repo uses custom engine creation to enable INT8 calibration and the model graph. The DeepStream sets the engine filename (/opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer/nvdsinfer_model_builder.cpp) according to the properties in the config file. Using engine-create-func-name, it sets the filename to:

suggestedPathName =
    modelPath + "_b" + std::to_string(initParams.maxBatchSize) + "_" +
    devId + "_" + networkMode2Str(networkMode) + ".engine";