search

OSU-NLP-Group / TravelPlanner

[ICML'24 Spotlight] "TravelPlanner: A Benchmark for Real-World Planning with Language Agents"
https://osu-nlp-group.github.io/TravelPlanner/
MIT License
215 stars 27 forks source link

Bug: DistanceMatrix tool in prompt didn't match the actual tool name which is GoogleDistanceMatrix #5

Closed JasonZhu1313 closed 6 months ago

JasonZhu1313 commented 6 months ago

In tool calling prompt, the model is instructed to use the DistanceMatrix tool. However, the actual tool name that is callable is GoogleDistanceMatrix. This mistake will cause the model to initially call DistanceMatrix and then fail with an observation stating that it needs to call the correct tool from the set {... GoogleDistanceMatrix ...} before making the right call. This is likely unintentional, and the model is not supposed to be aware of such a discrepancy.

Bug fix: https://github.com/OSU-NLP-Group/TravelPlanner/pull/4

hsaest commented 6 months ago

Hi Jason,

Thank you for your suggestion. I have merged your PR.

In the initial version code, we actually adopted the name "GoogleDistanceMatrix" in the prompt. We changed this in the released code, considering the consistency of the paper and code. Please be assured that this will not influence the experiment results shown in the paper.

Sorry for the confusion. Now, we consistently use "GoogleDistanceMatrix" in the code. Thank you for your insightful reminder.

If you have any further questions, please get in touch with me.

Best, Jian

JasonZhu1313 commented 6 months ago

Thanks for the detailed explanation, sounds good, great work again on releasing the benchmark