search

xlang-ai / Spider2

Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
https://spider2-sql.github.io
Apache License 2.0
157 stars 14 forks source link

Plans in the Spider2-lite #4

Closed ShayanTalaei closed 1 month ago

ShayanTalaei commented 1 month ago

Hi,

Did you pass the "plan" provided in the Spider2-lite as a part of the context in the benchmarked methods? e.g. agent, Dail-SQL, CodeS

Thank you!

yuxiaooye commented 1 month ago

Hi, sorry for the late reply! We are considering using "plan" as part of the context only in the oracle setting, since the majority of them are generated based on the gold SQL, which is not usable in the real text2SQL scenarios. Please refer to README of Dail-SQL for the performance improvements when using "plan".

For its detailed implementation in Dail-SQL and CodeS, see https://github.com/xlang-ai/Spider2/blob/98fc5c486a6b7c6202a118026f190ab21e480b03/spider2-lite/baselines/dailsql/prompt/PromptReprTemplate.py#L100 and https://github.com/xlang-ai/Spider2/blob/353004cf0916b398db9b3b753ce05ff793009ab4/spider2-lite/baselines/codes/utils/load_sft_dataset.py#L41