search

Psycoy / MixEval

The official evaluation suite and dynamic data release for MixEval.
https://mixeval.github.io/
196 stars 28 forks source link

Default Local Chat Tokenizer Padding Side "left" for multi batch inference #19

Closed carstendraschner closed 2 months ago

carstendraschner commented 2 months ago

This PR sets the default local_chat tokenizer padding to left to ensure consistent behavior across models. Without explicit specification, padding side can be incorrect, depending on the use case (e.g., training vs inference). This change is crucial for our Multi-batch inference in MixEval, which requires left padding.

For reference, Hugging Face's best practice and explanation can be found here: https://huggingface.co/docs/transformers/llm_tutorial#wrong-padding-side

Thank you for your efforts!

Psycoy commented 2 months ago

Hi,

The default padding side is indeed 'left'. All model classes inherits the base model class, which specifies that.

carstendraschner commented 1 month ago

Hi @Psycoy Thank you for your response. I am still facing inference issues and I guess they might come from the padding side component. Don't you overwrite the build tokenizer function which in the new implementation does not use padding side anymore as it is done in the ModelBase class? Or is Local chat part of @PhilSchmidt PR?

Local Chat https://github.com/Psycoy/MixEval/blob/4a572edf79e247ac202465025d54965ce277b49d/mix_eval/models/local_chat.py#L32

ModelBase https://github.com/Psycoy/MixEval/blob/4a572edf79e247ac202465025d54965ce277b49d/mix_eval/models/base.py#L57