Open sanderland opened 8 months ago
Hi,
Thanks for pointing this out! We will check these samples immediately and get back to you after processing.
These are some strings that are common in problematic prompts:
["Rewritten Prompt", "the given prompt and rewrite", "The Given Prompt"]
Thanks for your assistance!
I've meticulously inspected all these samples and found they are about prompt engineering. All models including the GPT-4 judge are not able to follow the instructions. Considering that these challenging instructions should be meaningful in examining models' instruction-following ability, we tend to manually rectify them rather than remove them from the dataset.
Currently, I am still striving to prompt the models, especially the GPT-4 judge, to understand the instructions, though little progress has been made. I'd appreciate it very much if anyone could help!
Gives
But there is no "given prompt". This seems to be an issue with several of the evol_instruct prompts. Also note that the completions for such samples include wild hallucinations, and ratings evaluating them as free of hallucinations.
In addition, even evol_instruct prompts that do include the prompt to be modified are often full of issues, with either the model or the evaluator interpreting it as a request to answer the original prompt.