search

macournoyer / neuralconvo

Neural conversational model in Torch
776 stars 347 forks source link

Question regarding encoder-decoder coupler #6

Closed cjmcmurtrie closed 8 years ago

cjmcmurtrie commented 8 years ago

Hi there, I have a question regarding the encoder-decoder coupler suggested in the examples here. My question is specifically regarding the decoder.

At the first time step, the decoder accepts the final embedding outputted by the encoder. After this first time step, the input for the next step is the decoder's output from the previous time step.

The way I have implemented a similar model in the past in Python, worked such that the final input from the encoder is input repeatedly to the decoder at each time step. My understanding is this is very useful to make sure the decoder is consistently addressing the encoded vector, rather than greedily generating sequences according to it's own output.

Have I understood the implementation correctly? If not, where is my understanding wrong? If so, what would be the best way to pass the encoded vector repeatedly to the decoder, rather than passing it's own outputs back into the next time step?

Thanks!

macournoyer commented 8 years ago

Feeding the output back to the decoder is what is described in the seq2seq paper: http://arxiv.org/abs/1409.3215 (what I understood from it). Which is why I implemented it this way.

You reasoning sounds very good to me. I think it would be worth testing...