Closed Ying1123 closed 3 months ago
This PR adds SGLang as a choice of model framework.
Tested by: adding following config to run.sh:
run.sh
MODEL_PATH="meta-llama/Llama-2-7b-chat-hf" TOKENIZER_PATH="meta-llama/Llama-2-7b-chat-hf" GPUS=1 MODEL_FRAMEWORK="sglang" SEQ_LENGTHS=(4096)
then run:
bash run.sh llama2-7b-chat synthetic
If interrupting your program cause SGLang hangs, kill SGLang by:
kill -9 $(ps aux | grep 'sglang' | grep -v 'grep' | awk '{print $2}')
This PR adds SGLang as a choice of model framework.
Tested by: adding following config to
run.sh
:then run:
If interrupting your program cause SGLang hangs, kill SGLang by: