search

LeapLabTHU / FLatten-Transformer

Official repository of FLatten Transformer (ICCV2023)
377 stars 21 forks source link

使用softplus对ReLU(query)进行缩放 #14

Closed YuhangGu4 closed 10 months ago

YuhangGu4 commented 10 months ago

models.flatten_pvt.py文件中的124-129行代码： kernel_function = nn.ReLU() scale = nn.Softplus()(self.scale) q = kernel_function(q) + 1e-6 k = kernel_function(k) + 1e-6 q = q / scale k = k / scale 您好，我对您的研究成果非常感兴趣。可以给我解释一下，您为什么要在ReLU(query)之后使用softplus进行缩放么，谢谢。

tian-qing001 commented 10 months ago

Hi @YuhangGu4, we use Softplus to ensure the non-negativity of scale. We did not use Softplus for query and key.

YuhangGu4 commented 10 months ago

了解了，谢谢您的答复。