Use of ONNX model in C++ (ONNX Runtime C++ API)

[adricostas]

Hello,

Has anybody tried to use these ONNX models on C++? I'm trying to do so but I'm getting a Segmentation Fault during the inference. The model is loaded without any problem and the input shape seems to be ok (1,3,288,512). In python the inference is working. The code is the following: main.cpp

#include "lightglue.h"
#include "extractor.h"
#include <opencv2/opencv.hpp>
#include <vector>

int main() {
    // Usage example

    std::string extractor_model_path = "/data/LightGlue-ONNX/weights/disk.onnx";

    std::vector<int> resize = {512};
    bool grayscale = false;
    std::pair<cv::Mat, std::vector<float>> image0 = loadImage("/images/000001.jpg", grayscale, resize);

    DiskRunner extractor_runner(extractor_model_path);
    extractor_runner.run(image0.first);

    return 0;
}

DiskRunner class (I have divided LighGlue class into two classes: one for the extractor and one for the matcher)

DiskRunner::DiskRunner(const std::string& extractor_path, bool use_cuda) {
    Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "DiskRunner");
    Ort::SessionOptions session_options;

    if(use_cuda){              
        OrtCUDAProviderOptions cuda_options{};
        cuda_options.device_id = 0;
        session_options.AppendExecutionProvider_CUDA(cuda_options);        
    } 

    session= std::make_unique<Ort::Session>(env, extractor_path.c_str(), session_options);

    Ort::AllocatorWithDefaultOptions allocator;

    const size_t num_input_nodes = session->GetInputCount();
    this->input_node_names.reserve(num_input_nodes);  

    // iterate over all input nodes
    for (size_t i = 0; i < num_input_nodes; i++) {
        auto input_name = session->GetInputNameAllocated(i, allocator);       
        this->input_node_names.push_back(strdup(input_name.get()));//strdup makes a copy
        this->input_node_dims= session->GetInputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape();

    }   

    const size_t num_output_nodes = session->GetOutputCount();
    this->output_node_names.reserve(num_output_nodes);

    for (size_t i = 0; i < num_output_nodes; i++) {
        auto output_name = session->GetOutputNameAllocated(i, allocator);
        this->output_node_names.push_back(strdup(output_name.get()));        

        this->output_node_dims= session->GetOutputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape(); 

    }

}

cv::Mat DiskRunner::run(cv::Mat image){

   // size_t inputTensorSize = vectorProduct(this->input_node_dims); //input_node_dims [1,3,-1,-1] dynamic shape 
    size_t inputTensorSize = image.total();
    std::vector<float> inputTensorValues(inputTensorSize);

    for (int i=0; i<image.dims; ++i)
        this->input_node_dims[i] = image.size[i];

    inputTensorValues.assign(image.begin<float>(),
                         image.end<float>());

    auto memory_info = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtDeviceAllocator, OrtMemType::OrtMemTypeCPU);
    auto input_tensor = Ort::Value::CreateTensor<float>(memory_info, inputTensorValues.data(), inputTensorSize,
                                                             this->input_node_dims.data(), this->input_node_dims.size());

    auto output_tensors = session->Run(Ort::RunOptions{nullptr}, this->input_node_names.data(), &input_tensor, this->input_node_names.size(), 
    this->output_node_names.data(), this->output_node_names.size());

    return cv::Mat();

}

The auxiliar functions:

cv::Mat readImage(const std::string& path, bool grayscale ) {
    int mode = grayscale ? cv::IMREAD_GRAYSCALE : cv::IMREAD_COLOR;
    cv::Mat image = cv::imread(path, mode);
    if (image.empty()) {
        throw std::runtime_error("Could not read image at " + path + ".");
    }
    if (!grayscale) {
        cv::cvtColor(image, image, cv::COLOR_BGR2RGB);
    }
    return image;
}

cv::Mat normalizeImage(cv::Mat& image) {
    cv::dnn::blobFromImage(image, image);  //HWC to NCHW  
    image.convertTo(image, CV_32FC1, 1.0 / 255.0);
    return image;
}

cv::Mat resizeImage(cv::Mat& image, const std::vector<int>& size, const std::string& fn, const std::string& interp ) {
    int h = image.rows;
    int w = image.cols;

    std::function<int(int, int)> func;
    if (fn == "max") {
        func = [](int a, int b) { return std::max(a, b); };
;
    }
    else if (fn == "min") {
        func = [](int a, int b) { return std::min(a, b); };

    }
    else {
        throw std::invalid_argument("Incorrect function: " + fn);
    }

    float scale;
    int h_new, w_new;
    if (size.size() == 1) {
        scale = static_cast<float>(size[0]) / static_cast<float>(func(h, w));
        h_new = static_cast<int>(round(h * scale));
        w_new = static_cast<int>(round(w * scale));
    }
    else if (size.size() == 2) {
        h_new = size[0];
        w_new = size[1];
        scale = static_cast<float>(w_new) / static_cast<float>(w);
    }
    else {
        throw std::invalid_argument("Incorrect new size: " + std::to_string(size.size()));
    }

    int mode;
    if (interp == "linear") {
        mode = cv::INTER_LINEAR;
    }
    else if (interp == "cubic") {
        mode = cv::INTER_CUBIC;
    }
    else if (interp == "nearest") {
        mode = cv::INTER_NEAREST;
    }
    else if (interp == "area") {
        mode = cv::INTER_AREA;
    }
    else {
        throw std::invalid_argument("Incorrect interpolation mode: " + interp);
    }

    cv::resize(image, image, cv::Size(w_new, h_new), 0, 0, mode);
    return image;
}

std::pair<cv::Mat, std::vector<float>> loadImage(const std::string& path, bool grayscale,
    const std::vector<int>& resize , const std::string& fn , const std::string& interp ) {
    cv::Mat img = readImage(path, grayscale);
    std::vector<float> scales = { 1.0f, 1.0f };
    if (!resize.empty()) {
        resizeImage(img, resize, fn, interp);
        scales = { static_cast<float>(img.cols) / static_cast<float>(resize[1]),
                   static_cast<float>(img.rows) / static_cast<float>(resize[0]) };
    }
    return std::make_pair(normalizeImage(img), scales);
}

cv::Mat rgbToGrayscale(cv::Mat& image) {
    cv::Mat result;
    cv::cvtColor(image, result, cv::COLOR_BGR2GRAY);
    return result;
}

Any idea ?

Thanks

[fabio-sim]

Hi @adricostas , thank you for your interest in LightGlue-ONNX!

Likewise, I've been testing the models using ONNXRuntime Python. However, your code seems awfully similar to this issue: https://github.com/microsoft/onnxruntime/issues/5630, in which another user also gets a segfault in C++ despite the model working fine in Python.

I'm assuming you have session as a member attribute of the DiskRunner class, which unfortunately outlives the Ort::Env env that was declared as a local variable in the constructor and subsequently destroyed at the end. The aforementioned issue suggests to make it a member of the class. Here is another issue mentioning the exact same thing: https://github.com/microsoft/onnxruntime/issues/5320. Here is a stackoverflow answer suggesting to declare env as static in order to extend its scope: https://stackoverflow.com/questions/62787710/how-can-i-fix-onnxruntime-session-run-problem

I could be totally wrong though, since I'm not well-versed in the C++ ONNXRuntime. If anyone more knowledgeable has a fix, please feel free to point it out.

I hope you find this helpful!

[adricostas]

Hi @fabio-sim,

You were right! Now it is working. Thank you for helping me even when the issue was not finally related with lightglue itself.

[antithing]

@adricostas Hi! Would you be able to share your working c++ inference code and models? Thank you!

[fxlong]

@adricostas Hi! Would you be able to share your working c++ inference code and models? Thank you!

[OroChippw]

@adricostas Hi! Would you be able to share your working c++ inference code and models? Thank you!

If you don't mind, you are welcome to use my repository https://github.com/OroChippw/LightGlue-OnnxRunner, currently supporting end-to-end model inference of SuperPoint and DISK ，Support for decoupling Superpoint/DISK and LightGLue will be provided in the future.If you have any suggestions or bugs, feedback is welcome.😊

[OroChippw]

@adricostas Hi! Would you be able to share your working c++ inference code and models? Thank you!

If you don't mind, you are welcome to use my repository https://github.com/OroChippw/LightGlue-OnnxRunner, currently supporting end-to-end model inference of SuperPoint and DISK ，Support for decoupling Superpoint/DISK and LightGLue .If you have any suggestions or bugs, feedback is welcome.😊