leaves162
commented
3 months ago Thank you for your interest in our work!
Regarding your questions, our answers are as follows:
- Semantic Correlation Computing
Our main idea is to use self-correlation to enhance the semantic correlation between local patches. Actually both the inner product and cosine can achieve this, but in order to complete the semantic segmentation module later, we introduce the clustering module. The infinite value range of the inner product caused trouble for our hyperparameter selection of clustering, so we introduced cosine. This allows us to maintain uniform hyperparameters for different datasets and improve versatility.
- Denoising
The core idea of denoising is: If the attention response of a local patch to itself is not the largest (that is, it is smaller than the response to the global patch), then it is likely to be noise. This is what we want to express in our paper. We are still working on and optimizing this method. We later discover that the attention weights of the cls token in the deep layers (not the last layer, which will be mentioned later) are all concentrated on the global patches, so the simple (attn weights - cls weights) can achieve the same screening idea, and is more concise in writing.
Regarding the use of the penultimate layer, in the end of A.1 in the supplementary material we said: The global patch phenomenon is slightly alleviated in the last layer due to the modal alignment objective function. So if we want the "noise filtering" operation in Denoising module to be more effective (that is, global patch phenomenon is easier to identify), we naturally need to use the penultimate layer.
Thank you again for your attention to this work. Hope my answers can help you.
------------------ 原始邮件 ------------------ 发件人: "leaves162/CLIPtrase" @.>; 发送时间: 2024年8月21日(星期三) 晚上11:01 @.>; @.***>; 主题: [leaves162/CLIPtrase] Questions about your implementation (Issue #1)
Thanks for your inspiring work.
Upon running your code, I noticed a couple of inconsistences from what was outlined in your paper:
Semantic Correlation Computing.
In your implementation, you derive the attention weights of the 12th layer in the following manner: image.png (view on web) image.png (view on web) It appears, however, that this approach is not explicitly mentioned in the original paper. I am uncertain whether I may have misinterpreted your paper. Could you kindly explain why you genrerate the attention weights in this specific manner?
Denoising
In the original paper, you indicate identifying noisy clusters using the criteria: image.png (view on web)
From my understanding, w_{i,i} represents self-correlation. Yet, in your implementation: image.png (view on web) You subtract the attention score of the "cls" token and other patches, which appears to deviate from the paper's statement. Could you kindly explain this for me? Additionally, I noted that you use the attention weights of the penultimate layer instead of the last layer for DBSCAN.
Could you assist me in clarifying the doubts I have mentioned above? Thank you.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>
hnczgytmp
Thanks for your inspiring work.
Upon running your code, I noticed a couple of inconsistences from what was outlined in your paper:
In your implementation, you derive the attention weights of the 12th layer in the following manner:
It appears, however, that this approach is not explicitly mentioned in the original paper. I am uncertain whether I may have misinterpreted your paper. Could you kindly explain why you genrerate the attention weights in this specific manner?
In the original paper, you indicate identifying noisy clusters using the criteria:
From my understanding, w_{i,i} represents self-correlation. Yet, in your implementation:
You subtract the attention score of the "cls" token and other patches, which appears to deviate from the paper's statement. Could you kindly explain this for me? Additionally, I noted that you use the attention weights of the penultimate layer instead of the last layer for DBSCAN.
Could you assist me in clarifying the doubts I have mentioned above? Thank you.