Intuition behind the share-weighted learnable mask token

[nielsRocholl]

Hi!

In your paper, you mention that:

For the embedding of each masked point patch, we replace it with a share-weighted learnable mask token

Could you provide some intuition behind why using a shared token for all masked tokens works? I had initially assumed that every patch would require a unique learnable masked token, but this of course, as proven by your work, is not the case.

Thanks!

[Pang-Yatian]

Well,.. it is a common practice to use the share-weighted learnable token. I think each patch using a unique token would cause overfitting problems.