INT8 Calibrator Usage

[azad96]

Hi,

Are readCalibrationCache and writeCalibrationCache methods executed in the background? I've included the calibrator related files in my own project and created an instance as in yolo.cpp. What I added to my createEngine method is as follows

if (builder->platformHasFastInt8()){
    std::cout << "Using INT8" << std::endl;
    Int8EntropyCalibrator calibrator(maxBatchSize, 
                                    "../calibration/calibration_images.txt", 
                                    "../calibration/images/", 
                                    "../calibration/yolo-calibration.table", 
                                    3*INPUT_H*INPUT_W, INPUT_H, INPUT_W, INPUT_BLOB_NAME);
    builder->setInt8Mode(true);
    // config->setFlag(nvinfer1::BuilderFlag::kINT8);
    builder->setInt8Calibrator(&calibrator);
    // config->setInt8Calibrator(&calibrator);
}

However, no calibration table is generated after the execution. Moreover, it seems like the calibration process never happens because when I enabled INT8 calibration, execution time does not change. Thanks in advance.

[azad96]

I've moved the calibrator instance outside of the if clause. Moreover, you should only set the IBuilderConfig's flag when using createBuilderConfig(). Calibration, otherwise, does not start for some reason. Therefore, you should use either of the following to start the calibration.

auto config = builder->createBuilderConfig();
config->setMaxWorkspaceSize(1 << 30);
builder->setMaxBatchSize(maxBatchSize);

Int8EntropyCalibrator calibrator(maxBatchSize, 
                                "../calibration/calibration_images.txt", 
                                "../calibration/images/", 
                                "../calibration/yolo-calibration.table", 
                                3*INPUT_H*INPUT_W, INPUT_H, INPUT_W, INPUT_BLOB_NAME);

if (builder->platformHasFastInt8()){
    config->setFlag(nvinfer1::BuilderFlag::kINT8);
    config->setInt8Calibrator(&calibrator);
}
nvinfer1::ICudaEngine* engine = builder->buildEngineWithConfig(*network, *config);

or

builder->setMaxWorkspaceSize((1 << 24));
builder->setMaxBatchSize(maxBatchSize);

Int8EntropyCalibrator calibrator(maxBatchSize, 
                                "../calibration/calibration_images.txt", 
                                "../calibration/images/", 
                                "../calibration/yolo-calibration.table", 
                                3*INPUT_H*INPUT_W, INPUT_H, INPUT_W, INPUT_BLOB_NAME);

if (builder->platformHasFastInt8()){
    builder->setInt8Mode(true);
    builder->setInt8Calibrator(&calibrator);
}
nvinfer1::ICudaEngine* engine = builder->buildCudaEngine(*network);

[alicera]

how to run the calibrator？ Thanks in advance.

[azad96]

@alicera, I normally implement models in tensorrt based on tensorrtx repo. However, this repo is also very similar to that. After adding the calibrator and calibrator related files in my project. I added the above calibrator object. Calibration starts automatically after that.