Open tymanman opened 3 years ago
I think it's because you have one vector of dimension hidden_size
for each possible x value, y value, w value and h value. Thus, each embedding layer has shape (max_2d_position_embeddings, hidden_size)
= (1024, hidden_size)
Although it's possible to represent those positions with only one scalar value for each, the model will likely generalize better if you use a (learnable) embedding layer instead :wink:
Describe Model I am using (UniLM, MiniLM, LayoutLM ...): I'm confused about the dimension of max_2d_position_embedding which is 1024 given in the code. I think the x,y or w,h is only one dimension, so how we can give a vector with dimension of 1024 to describe a point?