how to separate TrtNet init part and TrtNet doInference part for exporting it as a static libarary?

[dedoogong]

Hello! I'm trying to make it run with other pytorch models(so, it's python). So, I made some binding interface(libtrt.cpp/libtrt.h) between C++ and python. Then, I expected that I can call your TrtNet constructor and run trt->doInference from python code. But, when I call both in one extern function like

myLibTRTExternFunc(){
        **trtNet net("yolov3_int8.engine");**
        int outputCount = net.getOutputSize() / sizeof(float);
        unique_ptr<float[]> outputData(new float[outputCount]);
        ...
        ...
        **net.doInference(inputData.data(), outputData.get());**

}

it is called on every frames from a video file, which of course runs so slowly because of the engine init part.

So, I would like to separate those into two parts.

first, I made a init() function like ----------------------below function is in my libtrt.cpp --------------------------------

trtNet* trt_init(){
    printf("trt_init start\n");
    **trtNet net("yolov3_int8.engine");**
    printf("trt_init finished\n");
    if (&net== nullptr){/ 
        printf("Initialzing TRT failed\n");
    }
    else{
        printf("Initialzing TRT success : 0x%p\n",&net2);
        return ;
    }
}

vector<Bbox> trt_run(trtNet* net, cv::Mat img, int i) {
        printf("trt_run start\n");  
        int outputCount = net->getOutputSize() / sizeof(float); 
        unique_ptr<float[]> outputData(new float[outputCount]); 
        int classNum = 5;//
        list<vector<Bbox>> outputs; 
        cv::Mat img;
        cv::Mat img2;
        img2 = img.clone();
        vector<float> inputData = prepareImages(img); 
        **net->doInference(inputData.data(), outputData.get());**  
...
...
    }
}

----------------------below function is in my interfaceBindingModule.cpp --------------------------------

void run(){
     trtNet* trtnet=trt_init();
...
...
    trt_run((trtNet*)trtnet, img,i);
...
...     
}

it failed to call trtNet construnctor while engine deserealization(seg fault) even all input arguments are valid(

trtNet::trtNet(const std::string& engineFile)....
{...
...
    **printf("length %d   0x%p   0x%p\n",length, data.get(), &mTrtPluginFactory); **

}
---------------------------------- terminal -----------------------------------
*** deserializing
length 64947768 0x0x7fc591b86010 0x0x55bf545066b8
segmentation fault (core dumped) 

But it runs OK when I trtNet init part and trtNet doInference together in one function as I described firstly.

the Main problem is, the address of the trtNet instance becomes "null" after returning from Init function.

I have checked it is true clearly by printing the address of the instance(printed out like 0x123o98w in the init function, but "nil" after returned to another calling function.

So, I tried to make it "static" or "global"; that is, I wrote

extern trtNet nett("yolov3_int8.engine");

but all failed with the same result( seg fault during engine deserialization).

Can anyone give me any hint for passing trtNet instance between functions from separate source code files?

Thank you!

[dedoogong]

Originally tensorRT engine init(deserialization) and doInference(enqueue) parts are in one function. But if I want to call the init part only one time and then call doInference part several times for different input images, I need to separate those parts. First I thought I can create the trtNet instance like

<libTRT.cpp>
#include "libTRT.h"
trtNet* foo(){
trtNet net("myEningFile.engine");
}
void bar(trtNet* net,cv::Mat img){
net->doInference(img);
} 

<runTRT.cpp>
#include "libTRT.h"
main(){
void * net_handle=foo();
 ..do some openCV image read..
 bar((trtNet*)net_handle,readImg);
}

and I saw the returned net_handle value is "NULL" !! even thogh it had some pointer address value in the foo function(that's before return, it is valid but after return, the pointer value is gone!!)

So, I modified it to declare global variable,

trtNet net("myEngine.engine");

main(){

 ..do some openCV image read..
 bar([b]&net[/b],readImg);
}

it shows "seg fault" error while engine deserialization which runs OK in the previous cases.

[lewes6369]

I'm sorry your code is something hard to read. And I saw some efficiency problems or maybe some written mistakes on your code. As I see your bug may be that your instance is create in stack and return the object address in function stack. If you want to create an trtNet instance , try to create the factory class or function to create trtNet which the trtNet is created in heap. You can also try gdb to trace the functions.