[Question]: 使用llm进行量化时，PaddleSlim 和 PaddlePaddle 版本的对应关系有吗

liguodongiot commented 1 year ago

请提出你的问题

https://github.com/PaddlePaddle/PaddleNLP/tree/develop/llm#6-%E9%87%8F%E5%8C%96

请问在使用llm进行量化时，文档中 PaddleSlim 和 PaddlePaddle develop版本，但是安装了PaddlePaddle之后，并没有paddle.fluid，

但运行量化脚本的过程中 PaddleSlim 会使用 paddle.fluid。导致ModuleNotFoundError: No module named 'paddle.fluid'

使用llm进行量化时，PaddleSlim 和 PaddlePaddle 版本或者commitid的对应关系有吗？

或者有什么临时解决方案吗？

yangjq0630 commented 1 year ago

我是使用PaddlePaddle 2.5和PaddleSlim develop，能跑通，但是最终量化结果不理想，权重体积比原本大一倍，占用显存没变化，推理也更慢了，没研究清楚什么导致的

liguodongiot commented 1 year ago

@yangjq0630 您好，你是使用的GPTQ还是PTQ跑通的？

使用develop版本需要更改'paddle.fluid'为paddle.base。

from paddle.base.framework import IrGraph
from paddle.base.layer_helper import LayerHelper

yangjq0630 commented 1 year ago

@yangjq0630 您好，你是使用的GPTQ还是PTQ跑通的？

使用develop版本需要更改'paddle.fluid'为paddle.base。
from paddle.base.framework import IrGraph
from paddle.base.layer_helper import LayerHelper

两个都能跑通，就是效果不好

liguodongiot commented 1 year ago

@yangjq0630 您好，我安装了Paddle2.5.1之后，仍然会报错。你有遇到这个问题吗？我使用的是bloom模型进行量化。

看smoothquant原github项目，像是差了个act_scales，你看看你那个效果不好是不是这个导致的呢

[smooth ShiftSmoothHelpLayer] param: linear_95, before weight, abs_max: 1.8896484375
[smooth ShiftSmoothHelpLayer] param: linear_95, after weight, abs_max: 14.6953125
[2023-10-18 17:29:05,323] [    INFO] - ***** Smooth done *****
[2023-10-18 17:29:05,324] [    INFO] - ***** Running PTQ *****
[2023-10-18 17:29:05,387] [    INFO] -   Num examples = 114599
[2023-10-18 17:29:05,387] [    INFO] -   Total PTQ steps = 16
[2023-10-18 17:29:05,387] [    INFO] -   Pre device batch size = 8
[2023-10-18 17:29:05,388] [    INFO] -   Total Batch size = 8
I1018 17:29:09.167379 13522 eager_method.cc:140] Warning:: 0D Tensor cannot be used as 'Tensor.numpy()[0]' . In order to avoid this problem, 0D Tensor will be changed to 1D numpy currently, but it's not correct and will be removed in release 2.6. For Tensor contain only one element, Please modify  'Tensor.numpy()[0]' to 'float(Tensor)' as soon as possible, otherwise 'Tensor.numpy()[0]' will raise error in release 2.6.
Traceback (most recent call last):
  File "/home/guodong.li/workspace/code/PaddleNLP/llm/finetune_generation.py", line 452, in <module>
    main()
  File "/home/guodong.li/workspace/code/PaddleNLP/llm/finetune_generation.py", line 401, in main
    apply_ptq(quant_args, trainer, ptq_dataloader)
  File "/home/guodong.li/workspace/code/PaddleNLP/llm/quant.py", line 174, in apply_ptq
    act_scales[cur_name] = cur_layer.scales().numpy().tolist()
AttributeError: 'int' object has no attribute 'numpy'

github-actions[bot] commented 10 months ago

This issue is stale because it has been open for 60 days with no activity. 当前issue 60天内无活动，被标记为stale。

github-actions[bot] commented 10 months ago

This issue was closed because it has been inactive for 14 days since being marked as stale. 当前issue 被标记为stale已有14天，即将关闭。

PaddlePaddle / PaddleNLP