MeetKai / functionary

Chat language model that can use tools and interpret the results
MIT License
1.43k stars 110 forks source link

Prompt errors for Llama3.1 #264

Open el-hash-1 opened 2 months ago

el-hash-1 commented 2 months ago

Using the default code on huggingface page link, the prompt generated for tool calling is incorrect. This v3.2 model is based on Llama3.1 and it should be using this template defined in your repo

But this is the model output for the default tool and message on hf

image

Can someone please fix this?

And if you could provide a huggingface inference just like llama_cpp example that would be great!

jeffrey-fong commented 2 months ago

As mentioned here, we provide custom code to parse the raw response into JSON responses. Feel free to disable it by removing trust_remote_code=True when initializing your model and you will get the raw model responses.

el-hash-1 commented 2 months ago

As mentioned here, we provide custom code to parse the raw response into JSON responses. Feel free to disable it by removing trust_remote_code=True when initializing your model and you will get the raw model responses.

That's not what i meant, I am not just talking about the generated text, let me rephrase it.

The functionary-small-v3.2 model has the prompt format different from what it is supposed to be for llama3.1, If you check your tests/prompt_test_v3-llama3.1.txt

This is how it should look like if your base model is llama3.1 but it seems like the base model for your 3.2 version model is llama3 instead because the prompt matches to the tests/prompt_test_v3.llama3.txt

jeffrey-fong commented 2 months ago

v3.2 is finetuned with the prompt template in tests/prompt_test_v3.llama3.txt, not tests/prompt_test_v3-llama3.1.txt. The base model is llama3.1, not llama3. Llama3.1 can be finetuned using a different prompt template.