Create a separate position module for allen models. This position module will be used in the future for position embeddings that are based around targets within the text rather than the position embeddings that are within BERT or the Transformer XL models which are based on the position of the word within a text as they are performing language modelling which is a very different task. However the Transformer XL models does perform relative position encoding but it is relative to the token predicting which is always in front compared to a target that can have tokens that can be used either side of the token.
Create a separate position module for allen models. This position module will be used in the future for position embeddings that are based around targets within the text rather than the position embeddings that are within BERT or the Transformer XL models which are based on the position of the word within a text as they are performing language modelling which is a very different task. However the Transformer XL models does perform relative position encoding but it is relative to the token predicting which is always in front compared to a target that can have tokens that can be used either side of the token.
The positional weighting that needs to be added here is one from the following paper: Recurrent Attention Network on Memory for Aspect Sentiment Analysis