search

InternLM / lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
https://lmdeploy.readthedocs.io/en/latest/
Apache License 2.0
3.24k stars 288 forks source link

[Feature] Grammar/structured output support #1614

Open nidhoggr-nil opened 1 month ago

nidhoggr-nil commented 1 month ago

Motivation

Many agent and llm frameworks are starting to use structured output to make integration and parsing output work consistently.

I could not find any documentation if turbomind supports this or if it is in the works, but it seems very relevant.

Something like lm-format-enforcer would be very nice to have.

This should then be made to support tool calls via the openai API, so that turbomind becomes openai REST compatible with any framework.

lvhan028 commented 1 month ago

LMDeploy api_server is compatible with openai API Is https://lmdeploy.readthedocs.io/en/latest/serving/api_server.html#integrate-with-openai what you are looking for?

suc16 commented 2 weeks ago

1664 instructor

lvhan028 commented 2 weeks ago

@AllentDan is working on this feature