search

HKUSTDial / NL2SQL360

Official repository for the paper “The Dawn of Natural Language to SQL: Are We Fully Ready?” (VLDB'24)
https://arxiv.org/abs/2406.01265
MIT License
47 stars 4 forks source link

Can you advice my reproduction result? #5

Closed HeegyuKim closed 1 month ago

HeegyuKim commented 1 month ago

Hello! Thank you for sharing nice work!

I downloaded your published baseline results and evaluated them using the BIRD code. However, the BIRD benchmark result is different from the paper's.

I have been trying to solve this issue and I have noticed that my BIRD dev database filename is dev_20240627.zip. This different database version can be a reason for different reproduction results.

Could you share your dev dataset version? 스크린샷 2024-08-02 오후 6 52 00

BugMaker-Boyan commented 1 month ago

Thank you very much for your interest in our work!

Regarding the issue you raised, I have successfully reproduced and compared the results. The discrepancy is due to a recent update to the Dev dataset by the BIRD team, which means it is not the same version as the one used in our paper.

I have uploaded the BIRD Dev dataset we used in our paper to Google Drive. Here is the link: Google Drive

I hope this resolves your issue.

Additionally, we have recently released the NL2SQL360 Package, which provides an easy-to-use, multi-angle solution. Feel free to use it! If you have any other questions, feel free to discuss them with us!