The Result of WTQ on tapas_wtq_wikisql_sqa_masklm_large_reset model is only 39.04%

[lairikeqiA]

Thanks for releasing the new pre-trained model yesterday. I test the WTQ on tapas_wtq_wikisql_sqa_masklm_large_reset model. But the result is only 39.04% on the WTQ test dataset. What is the reason about that? My commend line is as follows: python3 run_task_main.py \ --task=WTQ \ --output_dir=/home/cjc/tapas-master/WTQ --model_dir=/home/cjc/tapas-master/tapas_wtq_wikisql_sqa_masklm_large_reset \ --init_checkpoint=/home/cjc/tapas-master/tapas_wtq_wikisql_sqa_masklm_large_reset/model.ckpt \ --bert_config_file=/home/cjc/tapas-master/tapas_wtq_wikisql_sqa_masklm_large_reset/bert_config.json \ --mode="predict_and_evaluate" --use_tpu=False --iterations_per_loop=5 --test_batch_size=4 --max_seq_length=512

[ghost]

You need to set the reset parameter and I am just realizing that that's not possible with the current run_task_main. I'll push a fix for that.

In the meantime, can you test that the tapas_wtq_wikisql_sqa_masklm_large model produces the reported accuracy?

[ghost]

I pushed the fix you can now run run_task_main.py with --reset_position_index_per_cell for the reset models.

[lairikeqiA]

Thanks for your reply. I tested WTQ on the tapas_wtq_wikisql_sqa_masklm_large model just now and got the reported accuracy on readme

[ghost]

Nice! Thanks for verifying.