search

Ffffffffire / HINormer

30 stars 5 forks source link

Experiment on vanilla Transformer #1

Closed khan-yin closed 1 year ago

khan-yin commented 1 year ago

hello,authors. I am curious about the experiment on vanilla Transformer for HGNN, can you tell me the details for training or show me some part of the code,thanks a lot😆

Ffffffffire commented 1 year ago

Thanks for your following! For vanilla Transformer, we implemented the vanilla Transformer referred to original paper(Multi-head self-attention + 2-layer FFN + 2 LayerNorm for a transformer layer). The architecture is same to HINormer with the same sampling process. And we discovered the L2-norm layer on the output and the bias terms of K,Q,V matrix(We set to False) have an impact on model performances.