SNLI_VE Inference Error

[sramshetty]

Hi, I've been trying to use the given Colab notebooks to perform the SNLI_VE task on the finetuned base model. However, after changing the task definition and creating my my sample, I get the following error: What could be going wrong here? Thanks in advance.

[JustinLin610]

Does the original code work well now? Any more details about how you make changes? Or you can share your colab notebook to us.

[sramshetty]

I haven't been able to get it to work. Here is the notebook: https://colab.research.google.com/drive/1SnYR_nWVNEYxGfY90FFaxLSMez7m-ug4?usp=sharing. Thanks.

I basically just change the task, model, and instruction components.

[sramshetty]

I guess a simpler question would be, how would you recommend one to run inference on the fine-tuned base model on snli-ve? Thank you for the help!

[sramshetty]

@JustinLin610 I tried a couple things, first tried to increase max steps in the config, but ran into:

RuntimeError                              Traceback (most recent call last)
[<ipython-input-10-a5275c952a04>](https://localhost:8080/#) in <module>
     19 with torch.no_grad():
     20   # hypos = task.inference_step(generator, models, sample, constraints=task.valid_answers_list)
---> 21   hypos = task.inference_step(generator, models, sample)
     22   tokens1, bins1, imgs1 = decode_fn(hypos[0][0]["tokens"], task.tgt_dict, task.bpe, generator)
     23   tokens2, bins2, imgs2 = decode_fn(hypos[0][1]["tokens"], task.tgt_dict, task.bpe, generator)

7 frames
[/content/OFA/models/ofa/unify_transformer.py](https://localhost:8080/#) in extract_features_scriptable(self, prev_output_tokens, code_masks, encoder_out, incremental_state, full_context_alignment, alignment_layer, alignment_heads)
   1509             self_attn_bias = self_abs_pos_bias.clone()
   1510             if code_masks is None or not code_masks.any():
-> 1511                 self_attn_bias += self.get_rel_pos_bias(all_prev_output_tokens, idx).unsqueeze(0)
   1512             elif code_masks is not None and code_masks.all():
   1513                 self_attn_bias += self.get_image_rel_pos_bias(all_prev_output_tokens, idx).unsqueeze(0)

RuntimeError: The size of tensor a (1025) must match the size of tensor b (1024) at non-singleton dimension 3

I then compared the SNLI-VE task against the Refcoco task and found that refcoco seems to atleast output results with the finetuned snli_ve base model. I think this may be because of different generators, but not entirely sure. So, I then tried to use refcoco's generator with constraints from the snli task, however I then get this error:

NameError                                 Traceback (most recent call last)
<ipython-input-42-d60fa7cb12fd> in <module>
     18 # Generate result
     19 with torch.no_grad():
---> 20   hypos = task.inference_step(generator, models, sample, constraints=task.valid_answers_list)
     21   tokens1, bins1, imgs1 = decode_fn(hypos[0][0]["tokens"], task.tgt_dict, task.bpe, generator)
     22 

5 frames
/usr/local/lib/python3.7/dist-packages/fairseq/search.py in <listcomp>(.0)
    243             elif self.representation == "unordered":
    244                 constraint_state = UnorderedConstraintState.create(constraint_tensor)
--> 245 
    246             self.constraint_states.append([constraint_state for i in range(beam_size)])
    247 

NameError: free variable 'constraint_state' referenced before assignment in enclosing scope

Not sure what I can do about that error, since it's part of fairseq's codebase. Would you have any ideas about how I should go about this, or is it possible? Thanks for any help in advance!