search

ARM-software / ComputeLibrary

The Compute Library is a set of computer vision and machine learning functions optimised for both Arm CPUs and GPUs using SIMD technologies.
MIT License
2.82k stars 774 forks source link

Int8 GEMM kernel #1006

Closed zhen-jia closed 1 year ago

zhen-jia commented 1 year ago

Hi All, I find that there is no int8 support in ACL. But I find there are some kernels seem implementing Int8 GEMM in https://github.com/ARM-software/ComputeLibrary/tree/main/src/core/NEON/kernels/arm_gemm.

I am wondering how to invoke those kernels, especially Int8 GEMM. Is there an example? Do you have a plan to support Int8 GEMM? Another question is what is 8-bit unsigned asymmetric quantized data type. Any documents? Thanks a lot.

Best, Zhen

morgolock commented 1 year ago

Hi @zhen-jia

I find that there is no int8 support in ACL. But I find there are some kernels seem implementing Int8 GEMM in https://github.com/ARM-software/ComputeLibrary/tree/main/src/core/NEON/kernels/arm_gemm.

There is support for int8 in ACL, you have to use the data type QASYMM8_SIGNED, see more in https://github.com/ARM-software/ComputeLibrary/blob/main/arm_compute/core/Types.h#L87

You can use the int8 kernels when calling to NEGEMMConvolutionLayer and the data type is QASYMM8_SIGNED , see https://github.com/ARM-software/ComputeLibrary/blob/main/arm_compute/runtime/NEON/functions/NEGEMMConvolutionLayer.h#L76

Another question is what is 8-bit unsigned asymmetric quantized data type. Any documents?

Please refer to https://www.tensorflow.org/lite/performance/quantization_spec

Hope this helps

zhen-jia commented 1 year ago

Thanks @morgolock for explanations. Let me read those.