search

xlang-ai / UnifiedSKG

[EMNLP 2022] Unifying and multi-tasking structured knowledge grounding with language models
https://arxiv.org/abs/2201.05966
Apache License 2.0
546 stars 58 forks source link

BLEC metric for SQL-to-Text #6

Closed awasthiabhijeet closed 2 years ago

awasthiabhijeet commented 2 years ago

Dear Authors,

Thanks for releasing this code.

I have a question regarding the BLEC metric for SQL-to-Text:

Does BLEC utilize a gold reference? Or is it just based on the predicted text and the input SQL? https://github.com/HKUNLP/UnifiedSKG/blob/0d47cd9382f0a3d17c70701056f399706ee3b698/third_party/BLEC/Spider/spider.py#L22

Thanks

ChenWu98 commented 2 years ago

Hi Abhijeet,

Thanks for double-checking! The short answer is: the SQL2Text BLEC does not utilize a gold reference. Here is a longer one: BLEC is a metric defined by Shu et al. (2021), and it requires hand-crafted templates for the logic matching process. As of Feb 5th, 2022, they have released the Logic2Text BLEC at https://github.com/chatc/BLEC, but not yet the SQL2Text BLEC. We contacted the authors last year, and they gave us the code generously, explaining that this code can only be used for the dev. and test splits proposed in their paper because currently they only have templates for these splits. Hope this information is helpful!

Thanks

awasthiabhijeet commented 2 years ago

I see. Thank you for the clarification.

Timothyxxx commented 2 years ago

Thanks for double-checking!