Closed damz closed 2 months ago
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 40.53%. Comparing base (
d0c62f8
) to head (88b93f4
).
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
The sentencepiece tokenizer is pretty slow, mostly because it allocates a ton. This PR removes some of the low-hanging fruit allocations.