search

ymcui / Chinese-LLaMA-Alpaca

中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)
https://github.com/ymcui/Chinese-LLaMA-Alpaca/wiki
Apache License 2.0
18.23k stars 1.86k forks source link

Adding --load_in_8bit option to inference_hf.py #755

Closed airaria closed 1 year ago

airaria commented 1 year ago

Description

Adding --load_in_8bit option to inference_hf.py

Usage Just add the --load_in_8bit to the launching command. For example:

python scripts/inference/inference_hf.py \
    --base_model path_to_merged_chinese_alpaca_plus \
    --with_prompt --interactive \
    --load_in_8bit