Open jexterliangsufe opened 9 months ago
谢谢提醒,我在复现代码中解码器输入H_dec和H'_enc顺序取反了;我认为H_dec是[BN, num_patch*2, embed_dim/2], H'_enc是[BN, num_patch, embed_dim/2], H_enc是[BN, num_patch, embed_dim],(论文中提到The Linear(·) layer is used to transform the hidden dimension from d_enc to d_dec),因此d_dec才是最终的dk,d_enc是d_dec的两倍。
按照论文中CA的公式,q应该是H_dec,k和v应该是H'_enc,不知道我理解的对不对
但是可以看到分母d_dec对应着d_k,我理解是因为d_dec=d_enc,所以作者随便写了一个上去?
[1]HUTFormer: Hierarchical U-Net Transformer for Long-Term Traffic Forecasting [2]Attention Is All You Need