search

mit-han-lab / lite-transformer

[ICLR 2020] Lite Transformer with Long-Short Range Attention
https://arxiv.org/abs/2004.11886
Other
596 stars 81 forks source link

about the global and local features in fig 3 #35

Closed sanwei111 closed 2 months ago

sanwei111 commented 3 years ago

as we know，the conventional attention module can capture features like fig 3.b（including diagonal and other positions）. THIS ability is its nature，BUT i JUST wonder that when we add a branch that can capture local features，the attention module can not capture feature like before，i.g，（including diagonal and other positions），while it just capture global feature！！！

wwwadx commented 2 years ago

Same question, how the model make sure that the attention layers capture the global information and the CNN layers capture local information with only one NLL loss? Have you figure it out?

as we know，the conventional attention module can capture features like fig 3.b（including diagonal and other positions）. THIS ability is its nature，BUT i JUST wonder that when we add a branch that can capture local features，the attention module can not capture feature like before，i.g，（including diagonal and other positions），while it just capture global feature！！！

sanwei111 commented 2 years ago

of course

---Original--- From: @.> Date: Tue, May 24, 2022 09:54 AM To: @.>; Cc: @.**@.>; Subject: Re: [mit-han-lab/lite-transformer] about the global and localfeatures in fig 3 (#35)

Same question, how the model make sure that the attention layers capture the global information and the CNN layers capture local information with only one NLL loss? Have you figure it out?

as we know，the conventional attention module can capture features like fig 3.b（including diagonal and other positions）. THIS ability is its nature，BUT i JUST wonder that when we add a branch that can capture local features，the attention module can not capture feature like before，i.g，（including diagonal and other positions），while it just capture global feature！！！

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

zhijian-liu commented 2 months ago

Thank you for your interest in our project. Unfortunately, this repository is no longer actively maintained, so we will be closing this issue. If you have any further questions, please feel free to email us. Thank you again!