search

sgl-project / sglang

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.
Apache License 2.0
2.79k stars 180 forks source link

Please add Phi3 support #407

Open Curiosity007 opened 2 months ago

Curiosity007 commented 2 months ago

Getting this error - 

router init state: Traceback (most recent call last):
  File "/home/ubuntu/sglang/python/sglang/srt/managers/router/manager.py", line 73, in start_router_process
    model_client = ModelRpcClient(server_args, port_args)
  File "/home/ubuntu/sglang/python/sglang/srt/managers/router/model_rpc.py", line 657, in __init__
    self.model_server = ModelRpcService().exposed_ModelRpcServer(
  File "/home/ubuntu/sglang/python/sglang/srt/managers/router/model_rpc.py", line 70, in __init__
    self.model_runner = ModelRunner(
  File "/home/ubuntu/sglang/python/sglang/srt/managers/router/model_runner.py", line 294, in __init__
    self.load_model()
  File "/home/ubuntu/sglang/python/sglang/srt/managers/router/model_runner.py", line 303, in load_model
    model_class = get_model_cls_by_arch_name(architectures)
  File "/home/ubuntu/sglang/python/sglang/srt/managers/router/model_runner.py", line 58, in get_model_cls_by_arch_name
    raise ValueError(
ValueError: Unsupported architectures: Phi3ForCausalLM. Supported list: ['CohereForCausalLM', 'DbrxForCausalLM', 'GemmaForCausalLM', 'LlamaForCausalLM', 'LlavaLlamaForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'QWenLMHeadModel', 'Qwen2ForCausalLM', 'StableLmForCausalLM', 'YiVLForCausalLM']
Gintasz commented 2 months ago

hmm, I just wanted to try this as well... sadly not supported yet.

Gintasz commented 2 months ago

@hnyls2002 what would be your thoughts? I'm using SGLang for bulk information extraction, so Phi3-mini would be very interesting to try since it's only 3.8B