get_chat_template using <|eot_id|> for tool use when <|eom_id|> should be used.

[William-Wildridge]

With llama 3.1 <|eom_id|> has been introduced to help support multi-turn reasoning.

End of message. A message represents a possible stopping point for execution where the model can inform the executor that a tool call needs to be made. This is used for multi-step interactions between the model and any available tools. This token is emitted by the model when the Environment: ipython instruction is used in the system prompt, or if the model calls for a built-in tool.

Currently get_chat_template is returning function calls with <|eot_id|> at the end of each request.

Docs here: https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/

[skmanzg]

I am planning to train Corpus with Llama 3.1 8b instruct and I noticed instruct model requires specific prompt. so I customized the function like this for Corpus training(SFTtraining)

def formatting_func(example):
    user_message = example["sentence"]
    formatted_prompt = f"<|begin_of_text|><|start_header_id|>user<|end_header_id|>{user_message}<|eom_id|><|start_header_id|>assistant<|end_header_id|>"
    return formatted_prompt

I haven't tried it yet with it but will it work? or will it be problematic?

[William-Wildridge]

I think this approach goes some way, and I'm curious if it's enough, but <|eot_id|> is not used in this example at all and I think omitting this could have a similar affect on model performance. I don't know if omitting one versus the other is preferrable, but it would be great to have a solution where only responses had eom and eot otherwise.

[danielhanchen]

Oh for tool use - yes <|eom_id|> should be used - I'll need to create a new chat template specifically it seems for tool usage

[asmith26]

Just wondering if anyone has successfully created a Google Colab notebook regarding finetuning tools for Llama 3.1? Thanks!

[danielhanchen]

Interesting idea - I might make a notebook for tool calling!