I want to extract the token embedding as shown in figure 11 of the paper.
However, when looking at the code, I see that the tokens are predicted by feeding the output feature map to a mlp whose last layer's dimension is 2003 (maybe number of tokens). Hence, the model do not learn the token embedding actually and we can't get the learned token embedding.
I want to extract the token embedding as shown in figure 11 of the paper.![image](https://user-images.githubusercontent.com/34507382/192301545-e36560a5-4e54-44f2-8114-5dd1d785b2f5.png)
However, when looking at the code, I see that the tokens are predicted by feeding the output feature map to a mlp whose last layer's dimension is 2003 (maybe number of tokens). Hence, the model do not learn the token embedding actually and we can't get the learned token embedding.
Am I missing something ?