search

PaddlePaddle / PaddleSlim

PaddleSlim is an open-source library for deep model compression and architecture search.
https://paddleslim.readthedocs.io/zh_CN/latest/
Apache License 2.0
1.56k stars 345 forks source link

The dtype of LinearQuanter's output is float #1878

Closed wanghaoshuang closed 5 months ago

wanghaoshuang commented 5 months ago 
import paddle
import numpy as np
from paddle.nn.quant.format import LinearQuanter

data = paddle.to_tensor(np.random.randn(2)).astype(paddle.float32)
int8_quanter = LinearQuanter(scales= 1.0, bit_length=8)
int16_quanter = LinearQuanter(scales= 1.0, bit_length=16)
quanted_data = int8_quanter(data)
print(quanted_data)
print(quanted_data.astype(paddle.int8))
quanted_data = int16_quanter(data)
print(quanted_data)
print(quanted_data.astype(paddle.int16))