Preprocessing code doesn't agree with toolllama_G123_dfs_train.json file in the provided data.zip

[HyunggyuJang]

Specifically, to convert OpenAI's function calling syntax to Llama compatible form, preprocessing is executed.

In the code, if the assistant message contains function_call property, it discards the content part, which is the part of Thought in Llama compatible syntax: https://github.com/OpenBMB/ToolBench/blob/3010778f5834fde71dc7658b4d51d1023affc21d/preprocess/preprocess_toolllama_data.py#L65

However, in toolllama_G123_dfs_train.json, the assistant message that contains both content and function_call, it convered to Thought: ${content}\nAction: ..., without discarding the content part.

[pooruss]

Hi, thanks for the question. The preprocess script is for the v1 version data, to process with v2 data (which contains both content and function_call), you can modify the script to add the content. We will soon update the preprocess script to process with v2 data.

[HyunggyuJang]

Thanks for your reply. Can I ask one more question? How could you manage to get a response from OpenAI model that contains both the content and function_call? To my knowledge, if the model decided to call function, the content part always null.

[zhoujz10]

Hi @pooruss Could you please release the preprocessing script for the v2 data? thanks!