search

OpenBMB / ToolBench

[ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.
https://openbmb.github.io/ToolBench/
Apache License 2.0
4.6k stars 397 forks source link

Mismatches between system prompt and parser #277

Open wchen-github opened 1 month ago

wchen-github commented 1 month ago

Here (https://github.com/OpenBMB/ToolBench/blob/5b5debedaf8b165ed5ecc30b210423e33f466870/toolbench/utils.py#L9C1-L15C26), the system prompt is specifying: "\nThought:\nAction\nAction Input:\n"

But in the parser (https://github.com/OpenBMB/ToolBench/blob/5b5debedaf8b165ed5ecc30b210423e33f466870/toolbench/inference/utils.py#L34), the patterns are: "Thought: ", "\nAction: " and "\nAction Input: "

The slight mismatches can sometimes cause parsing mistakes. Ideally, markup tags such as ... should be used but at the very least the mismatches should be made consistent.