search

openseg-group / OCNet.pytorch

Please choose the openseg.pytorch project for the updated code that achieve SOTA on 6 benchmarks!
MIT License
812 stars 128 forks source link

What's the meaning of the code snippet "sim_map = (self.key_channels**-.5) * sim_map" ? #19

Closed yu-changqian closed 6 years ago

yu-changqian commented 6 years ago

I find in the Base_OC_Module there is a magic code snippet sim_map = (self.key_channels**-.5) * sim_map, what's the meaning of it? And why?

PkuRainBow commented 6 years ago

@ycszen Please check the self-attention paper.

They call such operation scaled-product attention.