search

foundation-model-stack / fms-hf-tuning

🚀 Collection of tuning recipes with HuggingFace SFTTrainer and PyTorch FSDP.
Apache License 2.0
28 stars 48 forks source link

fix: do not add special tokens for custom tokenizer #279

Closed kmehant closed 3 months ago

kmehant commented 4 months ago

Description

When custom tokenizer is provided, the flow that adds special tokens to the tokenizer should be skipped.

Tests

Added one unit test.