search

MeetKai / functionary

Chat language model that can use tools and interpret the results
MIT License
1.43k stars 110 forks source link

Implement SGLang Server #277

Closed jeffreymeetkai closed 1 week ago

jeffreymeetkai commented 4 weeks ago

This PR is split into 3 parts:

  1. SGLang server (you can ignore the gsampling part)
  2. New unittest to make sure both vllm and sglang responses are exactly same as openai api's responses
    • it is similar to test_request_handling but launches the whole server with an actual model instead
    • It is not activated in Github CICD. It is activated only when needed locally
    • This unittest resulted in changes to prompt templates too to make sure responses match OpenAI's
  3. Change dependencies to pyproject.toml