Inference on peft trained falcon model gives nonsense

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

Apache License 2.0

37.12k stars 4.57k forks source link

Open jcarletgo opened 1 year ago

jcarletgo commented 1 year ago

I have a peft adapter model for a finetuned Falcon7b model, running it gives gibberish. The problem appears to be in get_generate_stream_function https://github.com/lm-sys/FastChat/blob/ae8abd20cbe1821100595d56d9ac4b137b31b53c/fastchat/model/model_adapter.py#L262 The model is detected as peft instead of falcon. Hardcoding is_falcon = True gives the expected output.

ilanazim commented 1 year ago

I experienced the same thing! Your fix works.

Before proposed change:

After: