Closed carstendraschner closed 2 months ago
Hi,
The score fluctuations comes from the below possible reasons:
Hi thanks for your response :) Could you please directly pin the line of file where I can find the system prompt you are referring to? thank you very much!
You can refer to the "How to use" section of their model card
Hi, I have a similar issue. I tried reproducing the llama3-8b-instruct results too and got lower results both for hard and full eval splits. I'm using gpt-3.5-turbo-0125 as an evaluator On the hard split: I had 38.75 instead of 45.6 On the full split: I had 73.7 instead of 75.0 I'm using no system prompt (I removed the pirate talk one). Are there any known issues that could cause this?
Hi,
I tried to reproduce the experiment results on a A100 while using Azure open AI API with GPT-35-Turbo-1106 as judge:
for Mistral7B it was fine for LLAMA8B it was: 0.39(mine) vs 0.46(yours). Do you have an Idea why it is this far off? I tried also with Nous Research version of LLAMA3 8B instruct
I faced the same with some other models. The system prompt had only very small influence when used