search

OSU-NLP-Group / TravelPlanner

[ICML'24 Spotlight] "TravelPlanner: A Benchmark for Real-World Planning with Language Agents"
https://osu-nlp-group.github.io/TravelPlanner/
MIT License
215 stars 27 forks source link

JSON data for query generation #21

Closed jiahao-mobi closed 4 months ago

jiahao-mobi commented 4 months ago

Hi, I am wondering whether the JSON input data that are used in generating the query (in build_query_generation_prompt function) will be released?

hsaest commented 4 months ago

Hi,

Thank you for your interest in our work.

Actually, we have released the JSON format data of the training set and validation set. However, in order to avoid possible cheating, we keep the JSON format data of the test set for evaluation.

jiahao-mobi commented 4 months ago

Thanks for reply, great work! Actually we are from a travel planning company and we have our own program based planner. We want to test the ability of our planner in your sandbox, and the planner we built takes structured data as input, that's why we want JSON for test set. It would be great if we can get a secret copy of the test JSON. We promise that we will not submit this result to the leaderboard, since we know it's an unfair comparison.

hsaest commented 4 months ago

HI,

Sorry to say that we avoid any form of test set release. Our query construction process is easy to follow and the scripts are provided, so in order to have similar data for use in the industry field, we encourage you to re-create some new queries.

jiahao-mobi commented 4 months ago

Hi, thank you for the response. Fully understand your concern!