search

facebookresearch / fairseq

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
MIT License
30.23k stars 6.38k forks source link

Sentence prediction task does not add BOS token #2376

Closed prihoda closed 4 years ago

prihoda commented 4 years ago

🐛 Bug

I noticed that even though the code expects the BOS token to be present in sentence prediction ( https://github.com/pytorch/fairseq/blob/master/fairseq/models/roberta/model.py#L273), it is actually not added by the sentence_prediction task. Is this expected behavior? I guess the task still works, but might produce unexpected results when different tokens appear at the first position.

To Reproduce

This is clear from the code, but can be validated by running test_roberta_sentence_prediction test in test_binaries.py in debug mode and inspecting the input dataset.

Expected behavior

BOS token should be added by sentence_prediciton task, same as in the masked_lm task: https://github.com/pytorch/fairseq/blob/master/fairseq/tasks/masked_lm.py#L126

Environment

N/A

prihoda commented 4 years ago

Never mind, I didn't realize you are supposed to use --init-token 0