About feature-to-token similarity map and token.

[songbingze]

Hello author! I am very interested in your work. Your work is excellent.

I have a few questions.

In the paper, why is the feature-to-token similarity map designed to choose columns 2 to m in Eq.(6)? And why does the token denote the selection of rows 2 to m in Eq.(6)? How does this process give high value to the first token? What would be the impact of making the sum of each row in the feature-to-token similarity map equal to 1?

Thank you again for your work.

[w1oves]

• The application of softmax ensures that the similarity of each row sums to 1.
• Some features do not need modification, meaning they are dissimilar to every token; in such cases, it would be reasonable for the total similarity of that row to be lower.
• After removing a token, the similarity for each row can range from 0 to 1.
• Removing any column can achieve a result similar to removing the first column; I merely chose the first column as it was the easiest option. The key is to remove one column so that the total sum of the similarity matrix S does not equal 1. This idea was inspired by Vision Transformers Need Registers. If you still have any further questions, please feel free to ask.

[w1oves]

Thank you very much for your support of Rein! If this project or my answer is helpful to you, please give me a star, which is very important to me. If you have any further questions, please feel free to ask.

[songbingze]

Of course, very nice work!

[w1oves]

Thank you very much!