search

google-research / tapas

End-to-end neural table-text understanding models.
Apache License 2.0
1.13k stars 217 forks source link

Can TAPAS be configured to abstain from making a prediction on a query? #109

Closed joshplasse closed 3 years ago

joshplasse commented 3 years ago

I have noticed that TAPAS, a large proportion of the time, returns a response with high confidence even when the query you ask is not answerable from the table you provide it with. E.g., if you have a table of transactions and you ask TAPAS about what you spent on a day where no transaction was made, TAPAS returns and incorrect prediction with very high confidence (many times > 0.999).

I am currently using the huggingface implementation of the model tapas-base-finetuned-wtq with the default config, but have tried other TAPAS models as well and this issue remains.

I have also fine-tuned the model, providing it with empty lists when the table does not contain sufficient information to answer a query. This has fixed this particular issue; however, it won’t be possible to anticipate all queries that may not be answerable from the table, so fine-tuning on a case-by-case basis does not seem practical.

Are there any recommendations for changes in the model and/or config files that would provide TAPAS with the ability to abstain, or at least be less confident, when making predictions for queries that cannot be answered from the provided table?

eisenjulian commented 3 years ago

Hi @joshplasse this is a very good question, in line with a general problem of over-confidence in ML models. While we haven't yet focused on the particular problem, your approach seems to be a good one. I would also recommend looking into using an ensemble, or potentially using several forward passes with dropout activated to simulate the same ensemble effect as described in https://arxiv.org/abs/1506.02142

We will also be releasing soon a system description for a submission on https://sites.google.com/view/sem-tab-facts where we train a binary classification model to predict query - table relevance by using noisy data, which can potentially be useful in this case.

eisenjulian commented 3 years ago

Closing this for the time being, feel free to reopen if needed. The system description I was referring to was published at https://arxiv.org/abs/2104.01099