search

opendatahub-io / vllm-tgis-adapter

vLLM adapter for a TGIS-compatible gRPC server.
Apache License 2.0
8 stars 11 forks source link

fix: add_special_tokens in tokenize #144

Closed rafvasq closed 1 month ago

rafvasq commented 1 month ago

66 made add_special_tokens true by default but its behaviour isn't replicated in /tokenize resulting in a different token count if ADD_SPECIAL_TOKENS is false. This PR fixes that by passing it in /tokenize and adds a test for the tokenize method.

I can follow this up with another test that compares the token count between the methods if required but otherwise this closes #141.

codecov-commenter commented 1 month ago

Codecov Report

Attention: Patch coverage is 72.72727% with 3 lines in your changes missing coverage. Please review.

Project coverage is 58.33%. Comparing base (896db8b) to head (891744f).

Files with missing lines Patch % Lines
tests/utils.py 62.50% 1 Missing and 2 partials :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #144 +/- ## ========================================== + Coverage 57.07% 58.33% +1.26% ========================================== Files 25 25 Lines 1554 1565 +11 Branches 259 261 +2 ========================================== + Hits 887 913 +26 + Misses 587 568 -19 - Partials 80 84 +4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.