Closed mjc14 closed 5 years ago
@mjc14 thank you for asking, it's very good question. Well I was confused the meaning of word "Bidirectional Transfomer" at first. But when I saw this description on paper, I knew that Bidirectional Transfomer = Transformer Encoder
which is used in Attention All You Need
We note that in the literature the bidirectional Transformer is often referred to as a “Transformer encoder” while the left-context-only version is referred to as a “Transformer decoder” since it can be used for text generation. -BERT Paper Section 3.1 Model Architecture
So bi-directional is meaning of, Transformer(self-attention) can see the context, both left and right side. Implementation of Transformer Encoder block is here https://github.com/codertimo/BERT-pytorch/blob/master/model/transformer.py.
Please feel free to ask more, if you have any question :) have a good day
@mjc14 if you dont have any further question, I'll close this issue in soon. thanx
hi, i am new user of pytorch, i want to know which part in your code can represent Bidirectional Transformers ? thanks .