spider test suite eval cannot evaluate fiben

[tshu-w]

Hi, when I use spider test suite eval to evaluate fiben, I got error from process_sql.py when passing golden sql, would you like to share you scripts to evaluate fiben?

Traceback (most recent call last):
  File "/project-path/src/vendor/test-suite-sql-eval/evaluation.py", line 938, in <module>
    evaluate(args.gold, args.pred, args.db, args.etype, kmaps, args.plug_value, args.keep_distinct, args.progress_bar_for_each_datapoint)
  File "/project-path/src/vendor/test-suite-sql-eval/evaluation.py", line 574, in evaluate
    g_sql = get_sql(schema, g_str)
  File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 557, in get_sql
    _, sql = parse_sql(toks, 0, tables_with_alias, schema)
  File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 509, in parse_sql
    from_end_idx, table_units, conds, default_tables = parse_from(toks, start_idx, tables_with_alias, schema)
  File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 390, in parse_from
    idx, table_unit, table_name = parse_table_unit(toks, idx, tables_with_alias, schema)
  File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 260, in parse_table_unit
    key = tables_with_alias[toks[idx]]
KeyError: '"listedsecurity"'

[lanwuwei]

Hi Tianshu,

You an follow PICARD code to call test-suite: https://github.com/ServiceNow/picard/blob/main/seq2seq/metrics/spider/spider_test_suite.py#L9 Since we only care about execution accuracy, it is not needed to parse the SQL.

Best, Wuwei

[lanwuwei]

Simply speaking, we will skip the examples that have execution errors for gold labels.