search

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Apache License 2.0
36.85k stars 4.54k forks source link

How can I use Multiple NPUs ? #3230

Open QuentinWang1 opened 6 months ago

QuentinWang1 commented 6 months ago

for example : python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2

maybe this following command need to be supported? python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device npu --num-npus 2 image

image

wangshuai09 commented 6 months ago

FYI https://github.com/lm-sys/FastChat/issues/3237