search

nvtransfer / RULER

This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
Apache License 2.0
646 stars 43 forks source link

Add SGLang backend #32

Closed Ying1123 closed 3 months ago

Ying1123 commented 4 months ago

This PR adds SGLang as a choice of model framework.

Tested by: adding following config to run.sh:

MODEL_PATH="meta-llama/Llama-2-7b-chat-hf"
TOKENIZER_PATH="meta-llama/Llama-2-7b-chat-hf"
GPUS=1
MODEL_FRAMEWORK="sglang"
SEQ_LENGTHS=(4096)

then run:

bash run.sh llama2-7b-chat synthetic

If interrupting your program cause SGLang hangs, kill SGLang by:

kill -9 $(ps aux | grep 'sglang' | grep -v 'grep' | awk '{print $2}')