search

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Apache License 2.0
36.63k stars 4.52k forks source link

Issues in Chatbot Arena: Input prompt gets clipped at certain token length for LLM model present #1291

Closed suraj-gade closed 1 year ago

suraj-gade commented 1 year ago

Hi,

I am facing issue while giving prompt to LLM models like oasst-pythia-12b, fastchat-t5-3b. I provided a prompt question along with context but I see that prompt is getting clipped off while providing input to model in Chat Arena. For ex. - I provided a prompt which of around 1010 token size to fastchat-t5-3b model in Arena (I understand that context length for fastchat-t5-3b is 2048) but the eventually the input taken by Arena tool was striped to around 672 tokens.

Can you please help me understand why this is happening or if I am missing something here.

merrymercy commented 1 year ago

We set a limit here https://github.com/lm-sys/FastChat/blob/ae2cb9349f1d9309f01cc90987371baedf1239ea/fastchat/constants.py#L10 to reduce the pressure of our server. If you host your own, feel free to change it.