Closed park12sj closed 7 months ago
Hi @park12sj ,
There is an issue that the accuracy is somewhat undermined when qt is done in the a setting(compute_dtype=int8, weight_dtype=int4)
You can try the combination of int4+bf16 and asymmetric quantization. The quantization of int8+bf16 will be integrated soon.
I wonder if my usage is wrong or if I can't expect performance improvement without qt.
Your usage is right!
We currently don't do much optimization for fp32 and fp16,
fp16 only saves bandwidth, and will convert to fp32 when calculating. So the performance does not reach the effect of fp16.
In the next stage, there will be kernel optimization for fp, which will bring performance benefits.
Hi, @zhenwei-intel
For bf16, I know that calculation is also supported by avx-512 or amx instruction set. Do you have any plans to support bf16 inference on non-qt llm runtime?
For example, it would be nice to be able to do data.toile with bf16 type below.(I haven't tried it because 'numpy' has no attribute 'bfloat16') https://github.com/intel/intel-extension-for-transformers/blob/caa715fb5a6df223b3ae9a49694e48c5e984c585/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_gptneox.py#L172-L180
Hi @park12sj ,
We will support int8(store) +bf16(computation) recently, it may take 1-2 weeks. The bottleneck of LLM on cpu is memory, so it's better to use qt-llm, if you can share your model, we can help to check the accuracy issue.
Thanks
int8 +bf16 is done
I'm experiencing two issues below. If there is any other way to improve the Inference speed while minimizing the loss of accuracy, please.
This seems to be because our model is basically fp16 weight, but itrex converts the model to fp32.
But, I hardcoded the ftype below to 1 and converted it to fp16, but it's also slower than baseline.(392ms -> 586ms)
I wonder if my usage is wrong or if I can't expect performance improvement without qt.
I'm sorry in advance that I can't show you the results of prompt and inference because they are sensitive information. What I'm curious about is whether qt-induced accuracy damage is common in small models of 5.8B.
I tried setting up below as a test, but I think it's a combination that's not yet supported
(compute_dtype=bf16, weight_dtype=int8)