How to deal with logits from position indices in the output layer?

[xiaoda99]

Dear guys,

I found that the position embeddings are concatenated with the word embeddings in the embedding layer. https://github.com/openai/finetune-transformer-lm/blob/bd1cf7d678926041e6d19193cab7e5cd8ce2fce6/train.py#L411 and the output layer also shares weights with this embedding layer, so it outputs logits for both word indices and position indices. https://github.com/openai/finetune-transformer-lm/blob/bd1cf7d678926041e6d19193cab7e5cd8ce2fce6/train.py#L176

My questions are:

1. During lm pretraining, did you mask out the logits from those position indices when computing the loss?
2. If I use the pretrained model as a LM to generate text, do I need to mask out these position indices' logits before softmax when sampling the next word?

BTW, I used the pytorch code ported by huggingface: https://github.com/huggingface/pytorch-openai-transformer-lm FYI, I also posted an issue there describing some details of my experiments: https://github.com/huggingface/pytorch-openai-transformer-lm/issues/36

• Da Xiao

[madisonmay]

@xiaoda99 in https://github.com/IndicoDataSolutions/finetune/blob/development/finetune/base.py#L544 masking the positional embeddings helped produce more reasonable generated text for us