Open YoungJaeChoung opened 3 months ago
Hello! Indeed, in text-to-SQL benchmarks, it is not uncommon to have multiple valid SQLs for a question. And typically, during annotation process, humans couldn't list out all possible SQLs. To resolve this issue, I would suggest take a look at evaluation methods other than Exact Match. For example, execution accuracy by BIRD-SQL.
Thanks for pointing this out anyways!
Hello! Indeed, in text-to-SQL benchmarks, it is not uncommon to have multiple valid SQLs for a question. And typically, during annotation process, humans couldn't list out all possible SQLs. To resolve this issue, I would suggest take a look at evaluation methods other than Exact Match. For example, execution accuracy by BIRD-SQL.
Thanks for pointing this out anyways!
Thank you for sharing the paper. I will read it. :)
I think some questions have alternative queries.
1.
file name: GeoNuclearData.json
question: 'How many nuclear power plants are
in preparation
to be used in Japan?'query: 'SELECT count(*) FROM nuclear_power_plants WHERE Country = "Japan" AND Status = "
Under Construction
"'possible query: "select count(*) from nuclear_power_plants where Country = 'Japan' and Status = '
Planned
'"question:
Where
is the first BWR type power plant built and located?query: SELECT
Longitude, Latitude
FROM nuclear_power_plants WHERE ReactorType = "BWR" ORDER BY ConstructionStartAt LIMIT 1possible query: select
Name, Country
from nuclear_power_plants where ReactorType = 'BWR' order by ConstructionStartAt limit 13.
file name: GeoNuclearData.json
question: 'How many PHWR are there today?'
query: "select count(*) from nuclear_power_plants where ReactorType = 'PHWR' and Status != 'Shutdown';"
possible query: 'SELECT count(*) FROM nuclear_power_plants WHERE ReactorType = "PHWR"'
4.
file name: GreaterManchesterCrime.json
question: 'Which area do most of the crimes happen?'
query: 'SELECT Location FROM GreaterManchesterCrime GROUP BY Location ORDER BY count(*) DESC LIMIT 1'
possible query: 'select LSOA from GreaterManchesterCrime group by LSOA order by count(*) desc limit 1;'
5.
file name: GreaterManchesterCrime.json
question: Where is the safest area?
query: SELECT Location FROM GreaterManchesterCrime GROUP BY Location ORDER BY count(*) LIMIT 1
possible query: select LSOA from GreaterManchesterCrime group by LSOA order by count(*) asc limit 1