SayedNadim
commented
3 years ago Hi, Sorry for the late reply. Using patches as convolution kernels for calculating the inner product similarities among patches has been used in previous studies, most prominently in 'generative image inpainting with contextual attention', on which, the attention mechanism is based. This operation essentially calculates the most similar patches for feature reconstruction. To understand the concept, you can think of a small patch convoluting with itself and other patches. As the values are similar in the case of convoluting with itself, the resultant value of the convolution will be bigger than convolution with other patches. Now, if we calculate the softmax of the whole feature map after convolution, the self-convoluted patch will have a higher class probability than the other patches. So, when we perform this operation for all patches in a feature map, we get the patches that have higher similarity values after performing softmax, as we are only calculating the most similar patches through this operation, which is essentially the same as calculating the inner product similarity. We want to have an explicit local attention mechanism as we need texture consistency between the inpainted region and background region. However, we also want to remove the mask value's influence while calculating the local similarity. That's why we have incorporated the mask pruning before calculating the local similarity. Hope this helps! Please let me know if you have any more questions. Cheers!
haolin512900
Thank you for your great poject! In your code model/Attention.py, how did the idea of using
prop_kernelsas the convolution kernel forconv_resultsin this class GlobalLocalAttention, or where it came from?and why you want to do this? I am lookforward your reply,good lucky to you !