search

mlc-ai / web-llm

High-performance In-browser LLM Inference Engine
https://webllm.mlc.ai
Apache License 2.0
13.79k stars 887 forks source link

Direct BNF support #634

Open Bainainai opened 4 days ago

Bainainai commented 4 days ago

I was able to find support for JSON schemas in the project, and it seems to actually get implemented under the hood by BNF. I was wondering if WebLLM could directly support BNF instead of reaching it in a roundabout way by first giving it a JSON schema. I tried to investigate to see if I could make it happen myself, but I got lost when I found references to mlc.grammar.BNFGrammarGetGrammarOfJSON and mlc.grammar.BNFGrammarFromSchema and couldn't figure out where those were defined. It seemed to me that they are supposed to come from the @mlc-ai/web-runtime package, but the TVM project that is referenced as its source code does not seem to have these functions in it.

CharlieFRuan commented 4 days ago

Just supported via ^0.2.76 after integrating with XGrammar: https://github.com/mlc-ai/web-llm/pull/635

Check out examples/json-schema's ebnfGrammarExample() for a full example