hey! I'm reading the code and I find that the atomic model architecture(self.input_transformers) is a huggingface BERT-encoder. It doesn't seem to add a attention_mask into the encoder, though padding the input. So is it necessary to add the attention_mask, and mask the same position in MSE-loss ?
actually, I think it's important NOT to mask it out. Since you want the model to learn to end the sentence by padding, so that it should know that after the END tokens, all the suffixes should be PAD.
hey! I'm reading the code and I find that the atomic model architecture(self.input_transformers) is a huggingface BERT-encoder. It doesn't seem to add a attention_mask into the encoder, though padding the input. So is it necessary to add the attention_mask, and mask the same position in MSE-loss ?