This PR fixes support for multi-EOS models (e.g. LLaMA 3.2, closes #287) and adds a new clean_special_chars: bool = False argument to model.attribute to support the cleaning of special characters from tokens in the out.source and out.target sequences using the native tokenizer.decode function provided by transformers.
Also adds GraniteForCausalLM, GraniteMoeForCausalLM and OlmoeForCausalLM to the model config.
Description
This PR fixes support for multi-EOS models (e.g. LLaMA 3.2, closes #287) and adds a new
clean_special_chars: bool = Falseargument tomodel.attributeto support the cleaning of special characters from tokens in theout.sourceandout.targetsequences using the nativetokenizer.decodefunction provided bytransformers.Also adds
GraniteForCausalLM,GraniteMoeForCausalLMandOlmoeForCausalLMto the model config.