search

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Apache License 2.0
36.56k stars 4.51k forks source link

Integrate vllm Error, TypeError: top_k must be an integer, got float #3501

Open alanhsu777 opened 1 month ago

alanhsu777 commented 1 month ago

When I use fastchat to integrate vllm, I get the error "TypeError: top_k must be an integer, got float". The reason is that vllm 0.5.5 has added a BUGFIX #7227, check top_k must be an integer。For fastchat, the default value of top_k in vllm_worker.py is set to -1.0.

1 2
surak commented 6 days ago

I can confirm. Changing from 1.0 to 1 fixes this.