Open Oliver-ss opened 1 year ago
I meet the same problem
Meet the same problem, I wonder it is the way we use Deepspeed wrong (I mean load fp16 ckpt and set dtype=int8 in init_inference) or it is just bugs in it ?
Hey @Oliver-ss (cc @molly-smith ) - were you able to debug this problem?
Similarly to @Tracin , I am also wondering whether we are missing some insights into how we are supposed to quantize models to int8
and run inference on them with DeepSpeed.
Hey @Oliver-ss (cc @molly-smith ) - were you able to debug this problem?
Similarly to @Tracin , I am also wondering whether we are missing some insights into how we are supposed to quantize models to
int8
and run inference on them with DeepSpeed.
I figured it out few days ago. There are few bugs in the code: weight should be transposed before quantized and quantizer group can not be too large. Not sure if they are fixed in newer version.
Thanks for the info @Tracin - can you share what % of speed up you saw (at bloom-3b, I guess?) when using int8
instead of fp16
?
@trianxy int8 actually cost more time since weights have to be dequantized
@trianxy int8 actually cost more time since weights have to be dequantized
Maybe you are referring to something different? I was thinking about inference speed (milliseconds per generated token) of the quantized model. In that case, as far as I understand, no dequantization is needed.
@trianxy I mean int8 costs more time for dequantization in Deepspeed for now on bloom. Besides, I saw int8 kernel inference with no dequantization on 176B models, which has the 1.0x - 0.5x speed compared to fp16, since input quantization costs time.
this issue is also being looking into here https://github.com/microsoft/DeepSpeed/issues/2876
Describe the bug Hi, I am trying to run a bloom3b model with 8bit quantization with two 2080Ti GPUs.
Because there exists no pre-quantized version, I simply change the
dtype
whendeepspeed.init_inference
but a shape mismatch error always occurs.I try to read the code and find that the int8 weight seems to nedd an extra permutation necessary according to the
ds_qkv_gemm
cuda implementation.https://github.com/microsoft/DeepSpeed/blob/fd1449c766f8dc0b0d77ef6389934d094b60a889/csrc/transformer/inference/csrc/pt_binding.cpp#L877
Here is the error:
Here is my code:
To Reproduce
deepspeed --num_gpu 2 run_deepspeed.py
Expected behavior A clear and concise description of what you expected to happen.
ds_report output torch version .................... 1.11.0+cu113 deepspeed info ................... 0.8.2+8be8c012, 8be8c012, master torch cuda version ............... 11.3 torch hip version ................ None nvcc version ..................... 11.4 deepspeed wheel compiled w. ...... torch 1.11, cuda 11.3
Screenshots If applicable, add screenshots to help explain your problem.
System info (please complete the following information):