Training template

[NicolasMejiaPetit]

I'm sorry if I am just blind and can't happen to find it, but I can not find the data format template used for training. I saw some references to "tool-llama-single-round" in the code (I thought that I might be the template but I cant find it). Or was no instruction template used and you trained the data in raw with an an eos stop token?

[pooruss]

Hi there, if you are looking for how the raw data is transformed into training data and what the training data format is, please refer to preprocess_toolllama_data.py. There are detailed steps for processing the messages with templates, and the templates are almost the same as the prompt to ChatGPT during data creation process, which can be found under https://github.com/OpenBMB/ToolBench/blob/master/toolbench/inference/Prompts/

[NicolasMejiaPetit]

Sorry, I don't think I asked the question the right way. When I'm training a model with like oobabooga or alpaca lora, I have to use a instruction format template like alpaca, vincuna, which are both in json files. Is there such template available for the data? Or is what you said still the same answer to this question?

[pooruss]

Yes, for example, the system prompt template is

You are AutoGPT, you can use many tools(functions) to do the following task.
First I will give you the task description, and your task start.
At each step, you need to give your thought to analyze the status now and what to do next, with a function call to actually excute your step.
After the call, you will get the call result, and you are now in a new state.
Then you will analyze your status now, then decide what to do next...
After many (Thought-call) pairs, you finally perform the task, then you can give your finial answer.
Remember: 
1.the state change is irreversible, you can't go back to one of the former state, if you want to restart the task, say "I give up and restart".
2.All the thought is short, at most in 5 sentence.
3.You can do more then one trys, so if your plan is to continusly try some conditions, you can do one of the conditions per try.
Let's Begin!
Task description: {task_description}

where the task description should be replaced with user query or instruction.

[NicolasMejiaPetit]

Is that the instruction template for fine tuning a model? A while back I converted the gorilla dataset into a alpaca like dataset (instruction input output) but it gave me awful results since the model didn’t know what information it needed to learn or output.

I found what I meant, this is the instruction template used to train a alpaca model:

{ "instruction,output": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n%instruction%\n\n### Response:\n%output%", "instruction,input,output": "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n%instruction%\n\n### Input:\n%input%\n\n### Response:\n%output%" }

And this is the one to train a model on vicuna dataset:

{ "instruction,output": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n\nUSER: %instruction%\n\nASSISTANT: %output%" }

Is there a training template like this for this dataset available?

(Asking because my computer is currently fine tuning a code 7b model for sql and python and I would love to finalize the training with this dataset.)