search

sgl-project / sglang

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.
Apache License 2.0
2.75k stars 177 forks source link

Multi-node Tensor Parallelism #550

Closed Ying1123 closed 1 week ago

Ying1123 commented 2 weeks ago

Similar to torchrun, enter command into each node with respective node-rank.

GLOO_SOCKET_IFNAME=eth0 python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-70B-Instruct --tp 16 --nccl-init-addr 172.16.5.82:20000 --nnodes 2 --node-rank 0
GLOO_SOCKET_IFNAME=eth0 python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-70B-Instruct --tp 16 --nccl-init-addr 172.16.5.82:20000 --nnodes 2 --node-rank 1