search

replicate / cog-llama-template

LLaMA Cog template
Apache License 2.0
307 stars 52 forks source link

Wrong prompt? #5

Closed YannDubs closed 1 year ago

YannDubs commented 1 year ago

Thanks for the great repo and for making the 70B model available!

From your website and from the code it seems that you are using the following prompt.

User: {instruction}
Assistant:

but the official LLaMA-2-chat prompt seems to be:

<s>[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>

{instruction} [/INST]

to understand the impact of the prompt I evaluated evaluations using AlpacaEval the outputs of replicate.run with the three following formatting before calling run

Given those results, I would consider using the default prompt. Note that the results for the default prompt are, if I understand correctly, still not correct because replicate.run will add User:. Is that indeed the case?

PS: I think https://github.com/a16z-infra/cog-llama-template/blob/fdcfc759159d16acf203f984833b97c15acb6f8b/config.py#L22 should be <s> instead of </s>

rajko-rad commented 1 year ago

Thank you so much for this detailed and thoughtful comment!! We have been working around the clock to update everything and should have it fixed soon!!

matt-bornstein commented 1 year ago

Thanks for this great summary @YannDubs. This is a known issue, and was a change made by the Meta team just prior to launch. We're updating all the models now to use the final prompt structure/ tokens. @joehoover

Mascobot commented 1 year ago

Solved. Thx Yann

Mascobot commented 1 year ago

Closing this issue now.