Is pos_embed NEED for every Block？

Sense-X / UniFormer

[ICLR2022] official implementation of UniFormer

Apache License 2.0

823 stars 111 forks source link

Is pos_embed NEED for every Block？ #118

Closed mywebinfo65536 closed 1 year ago

mywebinfo65536 commented 1 year ago

Hi, I found that there is a pos_embed module in CBlock、SABlock and SABlock_windows， is it need for every Block or just need position embeding in the first Block like CBlock ？

Andy1621 commented 1 year ago

Thanks for your question! Adding more pos_embed actually improves the accuracy. Since it is lightweight, I suggest adding it to every block.

mywebinfo65536 commented 1 year ago

ok， thanks for reply. I will try it later.