When using a server, one currently cannot use the model_overide_args which could be very useful, e.g. for rope scaling.
This is currently the sglang.launch_server.py:
import argparse
from sglang.srt.server import launch_server
from sglang.srt.server_args import ServerArgs
if __name__ == "__main__":
parser = argparse.ArgumentParser()
ServerArgs.add_cli_args(parser)
args = parser.parse_args()
server_args = ServerArgs.from_cli_args(args)
launch_server(server_args, None)
The model_overide_args would be the third argument to launch_server defaulting to None. Adding a small cli parser that allows arbitrary model args would be great, e.g.
python -m sglang.launch_server --model_overide_args.rope_scaling.factor 2 --model_overide_args.rope_scaling.type linear
When using a server, one currently cannot use the
model_overide_args
which could be very useful, e.g. for rope scaling.This is currently the
sglang.launch_server.py
:The
model_overide_args
would be the third argument tolaunch_server
defaulting toNone
. Adding a small cli parser that allows arbitrary model args would be great, e.g.