ai-spiderweb commented 1 year ago

Shifted Pauq

Custom splits(iid and target query length) for English and Russian versions of text-to-SQL tasks for measuring compositional generalization, based on original Spider text-to-SQL dataset.

Authors

Oleg Somov somov.ol.dm@gmail.com
Ekaterina Dmitrieva forkatdmi@gmail.com
Elena Tutubalina tutubalinaev@gmail.com

Implementation

Right now we do not make custom implementations in task files.

Usage

We only modified config.jsonnet files, therefore our evaluation should be run in the default way.

Checklist:

[x] I and my co-authors agree that, if this PR is merged, the code will be available under the same license as the genbench_cbt repository.
[x] Prior to submitting, I have ran the GenBench CBT test suite using the genbench-cli test-task tool.
[x] I have read the description of what should be in the doc.md of my task, and have added the required arguments.
[x] I have submitted or will submit an accompanying paper to the GenBench workshop.

vernadankers commented 1 year ago

Hello!

We are getting quite close to the deadline (September 1, 11:59PM anywhere on earth), so please make any final changes to your PR if required, and don't forget to submit your accompanying paper to Openreview via https://openreview.net/group?id=GenBench.org/2023/Workshop by September 1.

Good luck finalising your PR and paper, feel free to tag us if you have questions. Cheers, Verna On behalf of the GenBench team