Fix large amount of content skipped when chunk ends with closing apostrophe

ThatOpen / engine_web-ifc

Reading and writing IFC files with Javascript, at native speeds.

https://thatopen.github.io/engine_web-ifc/demo

Mozilla Public License 2.0

577 stars 170 forks source link

Fix large amount of content skipped when chunk ends with closing apostrophe #826

Closed szamocza closed 1 month ago

szamocza commented 1 month ago

When a text was being tokenized and the closing apostrophe falled at the end of the current content chunk, content part (equal to the size of the buffer -1) had been skipped.