Escape characters support - Githubissues

Dan-wanna-M / kbnf

A high-performance constrained decoding engine based on context free grammar in Rust

Other

34 stars 1 forks source link

Escape characters support #9

Closed Dan-wanna-M closed 2 months ago

Dan-wanna-M commented 3 months ago

[x] Very common ASCII escape characters
[x] Other ASCII escape characters
[x] \uXXXX basic unicode support
[x] \uXXXXXX full unicode support

Dan-wanna-M commented 3 months ago

One major problem is whether we want to support invalid UTF-8 bytes. It might be useful given how current BPE tokenizer works, but it requries significant refactoring of kbnf-syntax.

Dan-wanna-M commented 3 months ago

\uXXXX basic unicode support done in v0.1.6.

Dan-wanna-M commented 2 months ago

Fully supported(and actually support more features than this issue) in v0.1.7.