search

SCIR-HI / Huatuo-Llama-Med-Chinese

Repo for BenTsao [original name: HuaTuo (华驼)], Instruction-tuning Large Language Models with Chinese Medical Knowledge. 本草（原名：华驼）模型仓库，基于中文医学知识的大语言模型指令微调
Apache License 2.0
4.31k stars 422 forks source link

请问推理需要占用多少显存，我40GB运行infer.sh提示：CUDA error: out of memory #69

Closed Macvh closed 9 months ago

Macvh commented 10 months ago

python3.8 infer.py \ --base_model './llama-7b-hf' \ --lora_weights './lora-llama-med' \ --use_lora True \ --instruct_dir './data/infer.json' \ --prompt_template 'med_template'

在using lora的时候报错

image
Macvh commented 10 months ago

不知道是不是和bitsandbytes有关

image
s65b40 commented 10 months ago

单纯推理40GB应该是够用的，报错内容不是很完整，但看起来是cuda环境问题，请确定cuda环境已添加至环境变量且版本匹配

200109110057Wjy commented 9 months ago

请问你解决了吗，我也遇到同样的问题