Batch Size Greater Than 1 Causes "Error, not all required dimensions specified" in Custom TensorRT Implementation

pavelgrigoriev commented 3 weeks ago

Description:

I am working on a custom model using a class I named HypNet, which is based on the original YoloV8 implementation and utilizes TensorRT for inference. The model's input dimensions are reshaped to [40, 1, 1280]. When I attempt to run the model with a batch size of 2 (or any batch size greater than 1), I encounter the following error:

terminate called after throwing an instance of 'std::runtime_error' what(): Error, not all required dimensions specified.

However, if I set both optBatchSize and batchSize to 1, the error does not occur, and I can successfully obtain results.

Initialization with Batch Size 2 (Error Occurs):

HypNet::HypNet(const std::string& onnxModelPath, const HypNetConfig& config)
    : CLASS_NAMES(config.classNames) {
    Options options;
    options.optBatchSize = 2;
    options.maxBatchSize = 2;
    options.precision = config.precision;
    options.calibrationDataDirectoryPath = config.calibrationDataDirectory;

    if (options.precision == Precision::INT8 && options.calibrationDataDirectoryPath.empty()) {
        throw std::runtime_error("Error: Must supply calibration data path for INT8 calibration");
    }

    m_trtEngine = std::make_unique<Engine<float>>(options);
    auto succ = m_trtEngine->buildLoadNetwork(onnxModelPath, SUB_VALS, DIV_VALS, NORMALIZE);
    if (!succ) {
        throw std::runtime_error("Error: Unable to build or load the TensorRT engine.");
    }
}

std::vector<std::vector<cv::cuda::GpuMat>> HypNet::preprocess(const std::string& lineFilePath) {
    const int batchSize = 2;
    cv::Mat line = loadLine(lineFilePath);
    cv::Mat reshaped_line = line.reshape(40, {1, 1280});
    cv::cuda::GpuMat gpuImg;
    gpuImg.upload(reshaped_line);
    std::vector<cv::cuda::GpuMat> batchInput;
    batchInput.reserve(batchSize);

    for (int i = 0; i < batchSize; ++i) {
        batchInput.push_back(gpuImg.clone());
    }

    std::vector<std::vector<cv::cuda::GpuMat>> inputs{std::move(batchInput)};
    return inputs;
}

Initialization with Batch Size 1 (Works as Expected):

    HypNet::HypNet(const std::string& onnxModelPath, const HypNetConfig& config)
        : CLASS_NAMES(config.classNames) {
        Options options;
        options.optBatchSize = 1;
        options.maxBatchSize = 1;
        options.precision = config.precision;
        options.calibrationDataDirectoryPath = config.calibrationDataDirectory;

        if (options.precision == Precision::INT8 && options.calibrationDataDirectoryPath.empty()) {
            throw std::runtime_error("Error: Must supply calibration data path for INT8 calibration");
        }

        m_trtEngine = std::make_unique<Engine<float>>(options);
        auto succ = m_trtEngine->buildLoadNetwork(onnxModelPath, SUB_VALS, DIV_VALS, NORMALIZE);
        if (!succ) {
            throw std::runtime_error("Error: Unable to build or load the TensorRT engine.");
        }
    }

Additional Information:

Expected Output: When using a batch size of 2, I expect the output to be in the format of [2, 6, 1280]. Current Setup: Repository Version: 5.0 (for the implementation of TensorRT) Custom Model Input Dimensions: [40, 1, 1280] I have removed the cv::cuda::split(batchInput[img], input_channels); line from engine.h because my model has 40 channels. By doing this, I simulate having multiple batches:

    for (int i = 0; i < batchSize; ++i) {
        batchInput.push_back(gpuImg.clone());
    }

model.zip

thomaskleiven commented 2 weeks ago

Can you confirm that it works as expected if you export the ONNX model with a fixed batch_size of 2?

pavelgrigoriev commented 2 weeks ago

Can you confirm that it works as expected if you export the ONNX model with a fixed batch_size of 2?

Now the model inference is happening, but there is an error calling by the following line in engine.cpp:

if (input.size() != 1 || input[0].size() != 1)

The program terminates with the following error message: terminate called after throwing an instance of 'std::logic_error' what(): The feature vector has incorrect dimensions!

This occurs when applying the transformOutput function.

`std::vector

cyrusbehr / tensorrt-cpp-api

Batch Size Greater Than 1 Causes "Error, not all required dimensions specified" in Custom TensorRT Implementation #80