Closed hideaki-t closed 9 months ago
it parsed beyond it was asked then it returned something not asked.
e.g.
"binding" OR "あいうえお"
SQLite3 opens the tokenizer twice. one for binding and the other for あいうえお by shifting the start pointer and limiting input length.
binding
あいうえお
"binding" OR "あいうえお" * from here, length 7 bytes
"binding" OR "あいうえお" * from here and length 15bytes (3bytes char * 5)
but this package parsed the input text entierly, thus generated too many token including OR.
OR
fixed GH-36
it parsed beyond it was asked then it returned something not asked.
e.g.
SQLite3 opens the tokenizer twice. one for
bindingand the other forあいうえおby shifting the start pointer and limiting input length.but this package parsed the input text entierly, thus generated too many token including
OR.fixed GH-36