TigerResearch / TigerBot

TigerBot: A multi-language multi-task LLM
https://www.tigerbot.com
Apache License 2.0
2.24k stars 194 forks source link

fix exllamav2_hf_infer.py InferenceMode error #127

Closed yyq closed 1 year ago

yyq commented 1 year ago

推理量化模型的时候会遇到报错:

RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal tensor before doing inplace update.See https://github.com/pytorch/rfcs/pull/17 for more details.

新增一行代码:

with torch.inference_mode(mode=True):