Open ken-ando opened 2 years ago
This work introduces additional positional embedding for the number of tokens more than 512. https://github.com/nlpyang/PreSumm/blob/70b810e0f06d179022958dd35c1a3385fe87f28c/src/models/model_builder.py#L150-L154 But, this code doesn't seem to extend transformer. I think if the subsequent encoder does not have additional parameters, the shape will not match.
So, I guess the transformers automatically add the parameters of transformer, is this understanding correct?
This work introduces additional positional embedding for the number of tokens more than 512. https://github.com/nlpyang/PreSumm/blob/70b810e0f06d179022958dd35c1a3385fe87f28c/src/models/model_builder.py#L150-L154 But, this code doesn't seem to extend transformer. I think if the subsequent encoder does not have additional parameters, the shape will not match.
So, I guess the transformers automatically add the parameters of transformer, is this understanding correct?