search

tazz4843 / whisper-rs

Rust bindings to https://github.com/ggerganov/whisper.cpp
The Unlicense
607 stars 105 forks source link

Grammar Support #146

Open jcsoo opened 1 month ago

jcsoo commented 1 month ago

I've been working on a a native Rust GBNF parser that generates WhisperGrammarElementType arrays as needed for whisper-rs. It's not particularly complicated - about 40 lines of pest code and a few hundred lines to generate the arrays.

Would there be interest in adding this functionality to whisper-rs, perhaps as a separate subcrate?

I thought about whether this would be more useful as an external ggml grammar crate supporting both whisper.ccp and llama.cpp, but right now it seems like the implementation is copy-pasted between the two projects and there are no guarantees that they will not diverge, and I don't know if there is a single Rust llama implementation that it makes sense to integrate with.

This also brings up the question of what directions the dependencies should go. Right now my crate depends on whisper-rs in order to export WhisperGrammarElementType, but that doesn't seem ideal. An alternative could be to generate the raw buffers and pointers needed by whisper.cpp, perhaps as an opaque type, or to fill in caller-provided buffers.

Any thoughts?

jcsoo commented 1 month ago

In PR #145, I've updated set_grammar to take a simple Box<[u64]> with the following contents:

0: # of rules 1..N+1: index of rule N N+2..: grammar elements

This allows us to pass around a single self-contained item that doesn't have any self-referential pointers. It's also simple enough so that we don't need dependencies between whisper-rs and the grammar parser, and it can also be used in llama.cpp derivatives.