Add text-to-sql evaluation metrics

[shuaichenchang]

Add text-to-sql evaluation metrics

Overview

This PR is the first of three PRs to add text-to-sql evaluation metrics.

1. Add exact set match accuracy and execution accuracy metrics, and test files
2. Add loader and integration test
3. Add processors

Details

This PR adds metrics in explainaboard/metrics, and test functions in integration_tests/metric_test.py. The metrics use the code from explainaboard/third_party/text_to_sql_test_suit_eval for text-to-sql task evaluations.

[pfliu-nlp]

@shuaichenchang Regarding task-dependent PyPI libraries, maybe you can refer to this strategy to figure out a good way to introduce them?

[shuaichenchang]

@shuaichenchang Regarding task-dependent PyPI libraries, maybe you can refer to this strategy to figure out a good way to introduce them?

I am trying to add the required libraries for text-to-sql evaluation into setup.cfg. https://github.com/neulab/ExplainaBoard/pull/555/commits/8fa204391a69563518b47f61f96e2865cffdf0f5 But it seems not to work. Do you have any suggestions on how to add it? @pfliu-nlp @tetsuok

[tetsuok]

@shuaichenchang The configurations of the failing CI jobs need to be updated to make CI green. Namely, the following places:

• https://github.com/neulab/ExplainaBoard/blob/main/.github/workflows/ci.yml#L24
• https://github.com/neulab/ExplainaBoard/blob/main/.github/workflows/ci.yml#L43

CI jobs (unit-tests and integration-tests) are failing because these jobs aren't configured to install required packages with the extra, text_to_sql. In other words, these jobs need to run pip install ."[text_to_sql]" to install required packages for text-to-sql, but they currently run pip install . instead.

[tetsuok]

@shuaichenchang Any update on this PR? Would you check my comment? Let me know if you have any questions.

[shuaichenchang]

It seems I accidentally added others' commits to my PR when I merge the main into my branch. Now the PR has too many commits. Do you know how to undo them or I may need to create a PR just for the commits I submitted? @pfliu-nlp @odashi Sorry for the inconvenience.

[odashi]

@shuaichenchang How about:

1. withdraw this pull request
2. git checkout <your preferred commit> then git checkout -b <new tag>
3. propose a new pull request on the new tag

[shuaichenchang]

@shuaichenchang How about:

1. withdraw this pull request
2. git checkout <your preferred commit> then git checkout -b <new tag>
3. propose a new pull request on the new tag

@odashi Sounds great! I picked my commits and created a new branch and PR.