The attention maps of different Dilated Rates overlap with each other. In the figure, the same token is calculated only once, but in the code, the overlapped token is taken into account multiple times (because the output is a weighted sum of all Dilated Rates).
Would you have any comments on this? Is this expected behavior? @donglixp @shumingma
The attention maps of different Dilated Rates overlap with each other. In the figure, the same token is calculated only once, but in the code, the overlapped token is taken into account multiple times (because the output is a weighted sum of all Dilated Rates).
Would you have any comments on this? Is this expected behavior? @donglixp @shumingma