During lm pretraining, did you mask out the logits from those position indices when computing the loss?
If I use the pretrained model as a LM to generate text, do I need to mask out these position indices' logits before softmax when sampling the next word?
Dear guys,
I found that the position embeddings are concatenated with the word embeddings in the embedding layer. https://github.com/openai/finetune-transformer-lm/blob/bd1cf7d678926041e6d19193cab7e5cd8ce2fce6/train.py#L411 and the output layer also shares weights with this embedding layer, so it outputs logits for both word indices and position indices. https://github.com/openai/finetune-transformer-lm/blob/bd1cf7d678926041e6d19193cab7e5cd8ce2fce6/train.py#L176
My questions are:
BTW, I used the pytorch code ported by huggingface: https://github.com/huggingface/pytorch-openai-transformer-lm FYI, I also posted an issue there describing some details of my experiments: https://github.com/huggingface/pytorch-openai-transformer-lm/issues/36