Currently, tokenizers return Vec<String> where tokens are be slices of the input string. Moving to Vec<&str> would remove one memory copy and is likely to help with run time.
This should be possible with PyO3 0.7.0 (not yet released) that will allow using lifetime specifiers in pymethods.
rth
commented
4 years ago
Currently, tokenizers return
Vec<String>where tokens are be slices of the input string. Moving toVec<&str>would remove one memory copy and is likely to help with run time.This should be possible with PyO3 0.7.0 (not yet released) that will allow using lifetime specifiers in pymethods.