‘answer_coordinates’ and ‘answers’ predicted as '[]'

[1073521013]

Hello, I would like to ask a question. After the model fineturning is completed, answer_coordinates and answers are predicted [], but I can get the value directly using the pretrained model. Need to specify in advance, prediction for entailed and refuted is normal. As long as it is fine-tuned, no matter how many steps the ‘answer_coordinates’ and ‘answers’ predict result is always []. Hope gives some ideas. Thanks

[muelletm]

Thanks!

For TabFact we only train the classification layer you shouldn't use the ‘answer_coordinates’ or ‘answers’ field in the CSV (we might actually remove them at some point but for now treat them as undefined.)

[1073521013]

But I tried other models with pre-training. It's the same problem for WTQ,,WIKISQL,SQA. Thanks

[ghost]

Okay, that is a different story then.

Can you share how you fine-tuned the models? (Command line and the data you are using) Maybe also check the logs for any suspicious error.

[1073521013]

Because my task is to verify the relationship between the table and the statement and which cells can be proved, I chose TABFACT as my task, but I tried a pre-trained model trained on other datasets. Like what you said "For TabFact we only train the classification layer", but why I can get the ‘answer_coordinates’ and ‘answers’ value directly with the pre-trained model when predict? Fine-tuning command: python3 tapas/run_task_main.py \ --task="TABFACT" \ --output_dir="${output_dir}" \ --init_checkpoint="${tapas_data_dir}/model.ckpt" \ --bert_config_file="${tapas_data_dir}/bert_config.json" \ --mode="train" Data set source: https://sites.google.com/view/sem-tab-facts Thanks!

[SyrineKrichene]

Hi, sorry for the late answer,

When using --task="TABFACT" for fine-tuning the model cares only about the probability P(s|T) (Entailed / refuted). Thus, the model updates only the prediction (the score related to the [CLS] token) to reduce the error. All the other tokens' scores are not updated. In this case 'answer_coordinates’ and ‘answers’ are meaningless.

For some cases when using a particular pre-trained model checkpoint we can end up having meaningful 'answer_coordinates’ and ‘answers’. For example if we use a pre-trained model on SQA data then the pre-trained model already outputs 'answer_coordinates’ and ‘answers’ (as the task is to predict the answer cells) when calling predict.

On Tue, Dec 22, 2020 at 8:15 AM 1073521013 @.***> wrote:

Because my task is to verify the relationship between the table and the statement and which cells can be proved, I chose TABFACT as my task, but I tried a pre-trained model trained on other datasets. Like what you said "For TabFact we only train the classification layer", but why I can get the ‘answer_coordinates’ and ‘answers’ value directly with the pre-trained model when predict? Fine-tuning command: python3 tapas/run_task_main.py --task="TABFACT" --output_dir="${output_dir}" --init_checkpoint="${tapas_data_dir}/model.ckpt" --bert_config_file="${tapas_data_dir}/bert_config.json" --mode="train" Data set source: https://sites.google.com/view/sem-tab-facts Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/google-research/tapas/issues/98#issuecomment-749387954, or unsubscribe https://github.com/notifications/unsubscribe-auth/APARZOJ4W5WELJ22IQMGFGDSWBBRTANCNFSM4VC2D3GA .

[ManasiPat]

Does the value of answer_coordinates need to be specified, especially when we are implementing WTQ based complete weakly supervised task?