Landing on prompts

[MinaAlmasi]

Notes mostly for documentation purposes!

The new prompts have been selected and I'm in the midst of generating data across models and temperatures (currently 1, 1.5 and 2). See below for notes on this process.

New Task Prompts

#2.0 prompts
"dailymail_cnn_21": "summarize this in a few sentences: ",
"mrpc_21": "paraphrase this: ",
"stories_21": "write a story based on this: ", 
"stories_22": "continue the story: ", 
"dailydialog_21": "continue the conversation between A and B by writing a single response to the latest speaker. write only a concise response and nothing else: ", 

Principles behind the task prompts

I have tried to keep prompts as short and general as possible. That is, something that a user would write and not something that requires a lot of engineering, and the generations seem ok (with the some expected issues e.g., models producing "Sure! I will paraphrase this!"). The exception is dailydialog requires more specific instructions.

For the stories dataset, the prompts lead the models be repetitive and to start with "Title: ". The bsc thesis project used the prompt "continue the story" as it seemed to produce it less. As an addition, I have included the prompt "write a story based on this" to see how the models would fare with this. It was inspired from the instructions on the subreddit that the dataset was based on ("If you see a prompt you like, simply write a short story based on it.")

Removing the System Prompt from Llama2

We initially had a custom system prompt like this:

"llama2_chat": "You are an AI, but you do not deviate from the task prompt and you do not small talk. Never begin your response with 'Sure, here is my response: ' or anything of the like. It is important that you finish without getting cut off."

This custom prompt was created to see if we could get around the model producing completions beginning with e.g., "Sure, I'll paraphrase ...", but since it produces such artefacts regardless, I think it is best to remove it as the devs behind llama now recommend removing it per default due to performance issues (see their repository). This would also adhere to our principle of using these models as a regular user would do (HuggingFace also has system prompts as an extra feature not as a default on their llama interactive space)

Moving on from here

If you have any comments (@rbroc), please let me know. My current plan is to finish generating data with varying temperatures with the prompts above, and then we can manually inspect data and only regenerate/redo prompts if something looks very very weird.

UPDATE 14/03

We settled on all prompts with 21 suffix as mentioned in #51:

#2.0 prompts
"dailymail_cnn_21": "summarize this in a few sentences: ",
"mrpc_21": "paraphrase this: ",
"stories_21": "write a story based on this: ", 
"dailydialog_21": "continue the conversation between A and B by writing a single response to the latest speaker. write only a concise response and nothing else: ", 

[rbroc]

closing this as I think we have converged