bug: tokenization in Tokenize and Generate does not match if ADD_SPECIAL_TOKENS=False

opendatahub-io / vllm-tgis-adapter

vLLM adapter for a TGIS-compatible gRPC server.

Apache License 2.0

8 stars 11 forks source link

bug: tokenization in Tokenize and Generate does not match if ADD_SPECIAL_TOKENS=False #141

Closed tjohnson31415 closed 1 month ago

tjohnson31415 commented 1 month ago

The usage of add_special_tokens in Generate is not replicated in Tokenize. This means that the tokenization result/count will be different if ADD_SPECIAL_TOKENS=False.

Let's make sure to add a unit test and integration test for it if applicable!