search

OSU-NLP-Group / TravelPlanner

[ICML'24 Spotlight] "TravelPlanner: A Benchmark for Real-World Planning with Language Agents"
https://osu-nlp-group.github.io/TravelPlanner/
MIT License
215 stars 27 forks source link

Datasets problems #2

Closed SuperSupeng closed 7 months ago

SuperSupeng commented 7 months ago

I've noticed that all datasets are currently in file format. Are there any plans to access them through online searches in the future?

hsaest commented 7 months ago

Sorry, not sure what do you mean "online searches"? Could you explain in more detail? If you mean online loading, you can access our datasets through the huggingface, for example:

from datasets import load_dataset
data = load_dataset('osunlp/TravelPlanner','test')['test']

The "test" can be substitued by "train" and "validation".

drogozhang commented 7 months ago

Thanks for your interest!

Our benchmark is built upon a self-contained sandbox for planning evaluation, which is challenging enough for current LLMs and advanced planning strategies. Therefore, we don't plan to build an online version in the near future. However, we leave this option open and are happy to make an online version if we see remarkable progresses of language agents on the current environment.

Let us know if you have any other questions!

Best, Kai