Closed carstendraschner closed 2 months ago
Hi,
The default padding side is indeed 'left'. All model classes inherits the base model class, which specifies that.
Hi @Psycoy Thank you for your response. I am still facing inference issues and I guess they might come from the padding side component. Don't you overwrite the build tokenizer function which in the new implementation does not use padding side anymore as it is done in the ModelBase class? Or is Local chat part of @PhilSchmidt PR?
This PR sets the default local_chat tokenizer padding to left to ensure consistent behavior across models. Without explicit specification, padding side can be incorrect, depending on the use case (e.g., training vs inference). This change is crucial for our Multi-batch inference in MixEval, which requires left padding.
For reference, Hugging Face's best practice and explanation can be found here: https://huggingface.co/docs/transformers/llm_tutorial#wrong-padding-side
Thank you for your efforts!