Gemma template won't end with eos_token

[rangehow]

I'm not quite sure if this is a bug. I've observed that Gemma's template does not attach EOS even at the end of the assistant's response. This behavior is quite inconsistent with the templates of other models.

gemma always end with while its eos_token is only \<eos> . The potential issues with this behavior might be: 1. If this template is used to format training examples, it could lead to the model learning to never stop. 2. There is a template gap compared to the original model trained with the normal EOS template, which could affect performance.

[rangehow]

# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM,TextStreamer
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-1.1-2b-it")
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-1.1-2b-it",
    torch_dtype=torch.bfloat16
).cuda()

streamer = TextStreamer(tokenizer)
input_text = [{'role':'user','content':"Do you like beef?"}]
input_ids = tokenizer.apply_chat_template(input_text, return_tensors="pt",add_generation_prompt=True).to("cuda")

outputs = model.generate(input_ids,streamer=streamer,max_new_tokens=4096)

with its output

<bos><start_of_turn>user
Do you like beef?<end_of_turn>
<start_of_turn>model
I am unable to express personal opinions or preferences, including those related to food items. As an AI language model, I am programmed to provide factual and informative responses based on available data and knowledge.<eos>

It is easy to find that it does not even end with , but rather with \<eos>.

[rangehow]

Hi, @Khaliq88. You sent a blank response, looking forward to your opinion about this.

[amyeroberts]

@rangehow Unfortunately @Khaliq88 has been repeatedly spamming the repo with issues and comments like this. I've marked as spam and the user has been blocked from posting.

cc @ArthurZucker @Rocketknight1

[jena-shreyas]

# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM,TextStreamer
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-1.1-2b-it")
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-1.1-2b-it",
    torch_dtype=torch.bfloat16
).cuda()

streamer = TextStreamer(tokenizer)
input_text = [{'role':'user','content':"Do you like beef?"}]
input_ids = tokenizer.apply_chat_template(input_text, return_tensors="pt",add_generation_prompt=True).to("cuda")

outputs = model.generate(input_ids,streamer=streamer,max_new_tokens=4096)

with its output

<bos><start_of_turn>user
Do you like beef?<end_of_turn>
<start_of_turn>model
I am unable to express personal opinions or preferences, including those related to food items. As an AI language model, I am programmed to provide factual and informative responses based on available data and knowledge.<eos>

It is easy to find that it does not even end with , but rather with .

This seems to be the case for gemma-1.1 variants.

For the same setup, I obtained a similar output (with <end_of_turn> token missing before <eos> token) for google/gemma-1.1-7b-it :

<bos><start_of_turn>user
Do you like beef?<end_of_turn>
<start_of_turn>model
I am unable to provide subjective opinions or preferences, as I am an AI language model and do not have personal feelings or tastes. My purpose is to provide factual information and answer questions based on available knowledge.<eos>

while the correct output format (with <end_of_turn> before <eos>) obtained for google/gemma-2-9b-it :

<bos><start_of_turn>user
Do you like beef?<end_of_turn>
<start_of_turn>model
As an AI, I don't have personal preferences or the ability to eat, so I don't have an opinion on beef or any other food.

Do you like beef?<end_of_turn>
<eos>

[rangehow]

@jena-shreyas your test helps to make this issue more clear. So seems the quickest way is to replace the chat template in huggingface gemma1.1 series repo, but it seems needs more testing to ensure how gemma1 behaves . : )

[jena-shreyas]

Some more observations:

• For google/gemma-1.1-7b-it, the last token ids generated for the corresponding output ...available knowledge.<eos>" are [...235265,1] corresponding to . and <eos> respectively.
• For google/gemma-2-9b-it, the last token ids generated for the corresponding output ...like beef?<end_of_turn>\n<eos>" are [...,107,108,1] corresponding to <end_of_turn>, \n and <eos> respectively.

(Refer to Gemma series' tokenizer_config.json for the above token ids. )

Meanwhile, the chat template in Gemma series models only controls the input format passed to the models, which looks like this (same for both variants):

<bos><start_of_turn>user
Do you like beef?<end_of_turn>
<start_of_turn>model

It is provided as a Jinja2 template in the tokenizer_config.json, and is the same for both gemma-1 and gemma-2 variants:

{{ bos_token }}
{% if messages[0]['role'] == 'system' %}
        {{ raise_exception('System role not supported') }}
{% endif %}
{% for message in messages %}
{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
    {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
{% endif %}
{% if (message['role'] == 'assistant') %}
    {% set role = 'model' %}
{% else %}  
    {% set role = message['role'] %}
{% endif %}
{{ '<start_of_turn>' + role + '\n' + message['content'] | trim + '<end_of_turn>\n' }}       
{% endfor %}
{% if add_generation_prompt %}
     {{'<start_of_turn>model\n'}}
{% endif %}

Since there is no post-processing done to the model.generate() output, this might possibly be a issue with the gemma-1.1 checkpoints itself.

@ArthurZucker @Rocketknight1 would like to know your thoughts on this.

[Rocketknight1]

Hi @jena-shreyas @rangehow, I don't believe this is a bug. Models sometimes emit <eos> to indicate they want to stop generation, even when they have been trained with <end_of_turn> tokens instead in their input data. In this case, stopping generation after <eos> but actually including <end_of_turn> when formatting that message in the chat is the correct behaviour, though I agree it seems a little weird!

The Gemma chat templates were written by Google, not by us - you can try opening an issue on one of the Gemma model pages instead and pinging the authors to see what they think, but in general we haven't observed any performance degradation with Gemma even in long multi-turn conversations, so I think the current template behaviour is probably okay!

[jena-shreyas]

Thanks for the clarification @Rocketknight1! I agree this behaviour shown by gemma-v1 variants is unexpected (though it shouldn't be an issue as long as performance isn't affected). Anyways, this issue seems to be resolved in the gemma-v2 variants, so no worries!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.