SciSharp / LLamaSharp

A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.
https://scisharp.github.io/LLamaSharp
MIT License
2.72k stars 353 forks source link

Impossible Invalid GBNF Grammar Parsed #394

Open TheWorldEndsWithUs opened 11 months ago

TheWorldEndsWithUs commented 11 months ago

I am using the grammar below with OpenHermes-2.5-Mistral-7B-16k-GGUF.

root   ::= (object | array) endline?
endline ::= "<|im_end|>" ws
value  ::= object | array | string | number | ("true" | "false" | "null") ws

object ::=
  "{" ws (
            string ":" ws value
    ("," ws string ":" ws value)*
  )? "}" ws

array  ::=
  "[" ws (
            value
    ("," ws value)*
  )? "]" ws

string ::=
  "\"" (
    [^"\\] |
    "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes
  )* "\"" ws

number ::= ("-"? ([0-9] | [1-9] [0-9]*)) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? ws

# Optional space: by convention, applied in this grammar after literal chars when allowed
ws ::= ([ \t\n] ws)?

I read and parse the grammar in a few lines of code using llamasharp.

string grammarFileLocation = @"json grammar.gbnf";
string gbnf = File.ReadAllText(grammarFileLocation);
string gbnfTrimmed = rawGrammar.Trim();
var grammar = Grammar.Parse(gbnfTrimmed , "root");

When I use this grammar in inference params, on the second inference it fails with an assert "GGML_ASSERT: D:\a\LLamaSharp\LLamaSharp\llama.cpp:6649". You can find the assert here.

Is there a way I can get the grammar to run twice for inferences in quick succession?

TheWorldEndsWithUs commented 11 months ago

@martindevans suggested parsing the grammar again instead of re-using the parsed grammar and it worked! It's a temporary fix, but it operates as expected.