search

Dan-wanna-M / kbnf

A high-performance constrained decoding engine based on context free grammar in Rust
Other
40 stars 2 forks source link

Add `mask_token_ids` and `update_token_ids` API #7

Open Dan-wanna-M opened 4 months ago

Dan-wanna-M commented 4 months ago

Some users might want to mask an array of token ids(e.g. from top_p, top_k) rather than the whole logits. We probably need the caller to provide an output buffer considering how the FFI works.

Dan-wanna-M commented 2 months ago

This will essentially stop cache from functioning. Probably should be implemented after eager regex cache.