OSU-NLP-Group / TravelPlanner

[ICML'24 Spotlight] "TravelPlanner: A Benchmark for Real-World Planning with Language Agents"
https://osu-nlp-group.github.io/TravelPlanner/
MIT License
215 stars 27 forks source link

Train set的用处是什么,这个方法是怎么train 这些llm的 #19

Closed tttonytan closed 5 months ago

hsaest commented 5 months ago

Hi,

Thank you for your interest in our work.

We offer a training set to encourage further follow-up research based on agent learning (e.g., in-context learning, reinforcement learning) within the TravelPlanner. We've assessed baseline performance of current agent frameworks in a zero-shot setting, without training LLMs.

Feel free to contact us if you have further questions.

yananchen1989 commented 5 months ago

@hsaest good day. I find that the prompt in solo planning (planner_agent_prompt.format(text=ii['reference_information'], query=ii['query'])) is quite lengthy, usually around 10K even up tp 16K.

Therefore, can I say that, this is the primary reason why in the experiment results, for open-llm, only Mixtral-8*7B-MoE is used for testing. Since most other open LLMs cannot handle such long input context (except "lmsys/vicuna-7b-v1.5-16k"
"mistralai/Mistral-7B-Instruct-v0.2" (32K). Am I right ?

(for close-llms. open-35-turbo, 16K context length is accepted)

image

hsaest commented 5 months ago

@yananchen1989 True. We only choose these open-sourced models that support more than 8K context.