Casting Pipeline tensor to uint8 for Onnx model does not work

[thetonus]

Description

I am using a heavily quantized Onnx model where the Onnx Runtime is expecting a unit8 tensor. I think I transformed the pipeline tensor to uint8, but the error says the model input is int8 instead of uint8.

Other models work well. No issues with pipelines that spit out float32 model input.

_Note: The code posted below is Kotlin. For visual debugging, it should not be an issue for that conversion.

Expected Behavior

Casting pipeline tensor to uint8 should allow a uint8 model to be able to process the model input.

Error Message

[DEBUG] 2021-07-14 16:46:25.758 [main] Engine - Found default engine: MXNet
[WARN ] 2021-07-14 16:46:25.759 [main] SimpleRepository - Simple repository pointing to a non-archive file.
[DEBUG] 2021-07-14 16:46:25.759 [main] ModelZoo - Checking ModelLoader: ai.djl.localmodelzoo:model.onnx UNDEFINED [
        ai.djl.localmodelzoo:model.onnx:N/A {}
]
[DEBUG] 2021-07-14 16:46:25.760 [main] Resource - Preparing artifact: file:/Users/user/.cache/sparsezoo/b2d2739b-b30f-445d-b1da-4ed9f39e9c61/model.onnx, ai.djl.localmodelzoo:model.onnx:N/A {}
[DEBUG] 2021-07-14 16:46:25.760 [main] SimpleRepository - Skip prepare for local repository.
Loading:     100% |████████████████████████████████████████|
[DEBUG] 2021-07-14 16:46:26.122 [main] CudaUtils - cudart library not found.
[DEBUG] 2021-07-14 16:46:26.124 [main] LibUtils - Using cache dir: /Users/user/.djl.ai/mxnet
[DEBUG] 2021-07-14 16:46:26.124 [main] LibUtils - Loading mxnet library from: /Users/user/.djl.ai/mxnet/1.8.0-mkl-osx-x86_64/libmxnet.dylib
Exception in thread "main" ai.djl.translate.TranslateException: ai.djl.engine.EngineException: ai.onnxruntime.OrtException: Error code - ORT_INVALID_ARGUMENT - message: Unexpected input data type. Actual: (tensor(int8)) , expected: (tensor(uint8))

How to Reproduce?

const val IMAGE_DIMENSIONS = 320

fun createPipeline(): Pipeline {
    return Pipeline()
        .add(CenterCrop())
        .add(Resize(IMAGE_DIMENSIONS, IMAGE_DIMENSIONS))
        .add(ToTensor())
        .add { array ->
            array.toType(DataType.UINT8, false)
        }
}

val pipeline = createPipeline()

val criteria: Criteria<Image, DetectedObjects> = Criteria.builder()
    .optApplication(Application.CV.OBJECT_DETECTION)
    .setTypes(Image::class.java, DetectedObjects::class.java)
    .optProgress(ProgressBar())
    .optTranslator(
        YoloV5Translator
            .builder()
            .setPipeline(pipeline)
            .optSynsetUrl("https://mlrepo.djl.ai/model/cv/object_detection/ai/djl/pytorch/classes_coco.txt")
            .build()
    )
    .optModelPath(Paths.get("~/Downloads/sparse_model.onnx))
    .optEngine("OnnxRuntime")
    .build()

fun loadModel(): ZooModel<Image, DetectedObjects> {
    return criteria.loadModel()
}

fun predict(img: ai.djl.modality.cv.Image): DetectedObjects? {
    val model: ZooModel<Image, DetectedObjects> = loadModel()
    val predictor: Predictor<Image, DetectedObjects> = model.newPredictor() as Predictor<Image, DetectedObjects>
    return predictor.predict(img)
}

Steps to reproduce

val img: ai.djl.modality.cv.Image = ...
predict(img)

What have you tried to solve it?

Try casting in other parts of input

[frankfliu]

This is OnnxRuntime's limitation, OnnxJavaType doesn't have uint8 data type.

See: https://github.com/microsoft/onnxruntime/issues/6261

[thetonus]

@frankfliu Thanks. This is quite unfortunate though. This issue also affects float16 (half precision) values as well. 😞

[frankfliu]

@hammacktony I created PR in onnxruntime to add uint8 support: https://github.com/microsoft/onnxruntime/pull/8401