mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation
https://llm.mlc.ai/
Apache License 2.0
19.23k stars 1.58k forks source link

[Question] Does MLC_LLM MLCEngine have an equivalent API for `llm.generate` in VLLM or SGLang? #3034

Open pjyi2147 opened 4 days ago

pjyi2147 commented 4 days ago

I am trying to use the direct output of MLCEngine class but I do not have any clue how to get the output based on the MLC-LLM docs. Is there any similar API to MLCEngine such as the examples in MLC-LLM?

https://docs.vllm.ai/en/latest/getting_started/quickstart.html https://sgl-project.github.io/backend/offline_engine_api.html