search

akshayatam / machine-translation-with-retnet

A machine translation model using Retentive Network
0 stars 0 forks source link

encoder_output isn't used in the RetNet decoder. #2

Open Kimchangheon opened 9 months ago

Kimchangheon commented 9 months ago

Hi, thanks for sharing your code! I'm also implementing an encoder-decoder model with a similar structure to yours.

However, I'm confused about why the encoder_output isn't used in the RetNet decoder. In my understanding, it should be fed directly or subjected to a cross-attention mechanism, similar to the original Transformer architecture. However, in your code, the encoder_output seems to be unused entirely.

akshayatam commented 9 months ago

Thanks for this issue. I am working on the fix. Apparently, the RetNet decoder does not have an argument for taking in the embeddings (see this post for more information). Being not very well acquainted with the model, I am trying to fix this for a more complete translation model.