Hi! Thank you for the great project.
I read the paper "GPT understand, too!". I understand that if an input such as "The capital of Britain is [MASK]" is given, then only the context Britain and Target [MASK] are embedded in the pre-train embedding layer, and the rest of the tokens are embedded in the prompt-encoder.
=> h(The), h(capital), h(of), e(Britain), h(is), e([MASK]) (h means prompt-encoder, e means pre-trained embedding layer).
But when I checked the code, the input of the prompt-encoder was always the same as [0, 1, 2, 3, 4, 5].
Can anyone explain what I misunderstood?
sooftware
commented
3 years ago
Hi! Thank you for the great project.
I read the paper "GPT understand, too!". I understand that if an input such as "The capital of Britain is [MASK]" is given, then only the context Britain and Target [MASK] are embedded in the pre-train embedding layer, and the rest of the tokens are embedded in the prompt-encoder.
=> h(The), h(capital), h(of), e(Britain), h(is), e([MASK]) (h means prompt-encoder, e means pre-trained embedding layer).
But when I checked the code, the input of the prompt-encoder was always the same as [0, 1, 2, 3, 4, 5].
Can anyone explain what I misunderstood?
Thank you.