Saves around 1,300ms when ingesting 15k documents (averaging 1000 chars each).
We spend 1.5x the tokenization time just cloning the tokens, which should be avoidable but is not included in this PR because the lifetimes get hairy and the % that tokenization takes in the total compaction time is currently fairly low.
Summarize the changes made by this PR.
Improvements & Bug fixes
removes the mutex from the tokenizer, reducing overhead
simplifies types by removing a Box wrapper
Test plan
How are these changes tested?
Covered by existing tests.
Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?
Description of changes
Saves around 1,300ms when ingesting 15k documents (averaging 1000 chars each).
We spend 1.5x the tokenization time just cloning the tokens, which should be avoidable but is not included in this PR because the lifetimes get hairy and the % that tokenization takes in the total compaction time is currently fairly low.
Summarize the changes made by this PR.
Box
wrapperTest plan
How are these changes tested?
Covered by existing tests.
Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?
n/a