npu支持GPTQ量化导出吗 - Githubissues

hiyouga / LLaMA-Factory

Unify Efficient Fine-Tuning of 100+ LLMs

Apache License 2.0

25.52k stars 3.16k forks source link

npu支持GPTQ量化导出吗 #4594

Closed murray-z closed 5 days ago

murray-z commented 5 days ago

Reminder

[X] I have read the README and searched the existing issues.

System Info

Platform: Linux-5.4.0-26-generic-x86_64-with-glibc2.31
Python version: 3.10.14
PyTorch version: 2.1.0+cpu (NPU)
Transformers version: 4.41.2
Datasets version: 2.19.2
Accelerate version: 0.30.1
PEFT version: 0.11.1
TRL version: 0.9.4
NPU type: Ascend910ProB
CANN version: 8.0.RC1.alpha001
DeepSpeed version: 0.14.0

Reproduction

### model
model_name_or_path: Llama3-8B-Chinese-Chat
template: llama3

### export
export_dir: Llama3-8B-Chinese-Chat-gptq-bit4
export_quantization_bit: 4
export_quantization_dataset: gptq_100.json
export_size: 5
export_device: cpu
export_legacy_format: false

Expected behavior

执行导出命令，一直没有报错，也没有相关生成。

Others

No response

hiyouga commented 5 days ago

不支持