search

tensorflow / tensor2tensor

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
Apache License 2.0
15.5k stars 3.49k forks source link

Context for dialog models #913

Open JohannesTK opened 6 years ago

JohannesTK commented 6 years ago

There has been a lot of advancements recently in achieving context for dialog models through a separate context layer. Eg. HRAN or VHRED

I'm trying to see if it is achievable only with the transformer model by appending the whole context as an input sentence.

Meaning context:

Question Answer Question Answer

Would become:

Question (this holds the two previous sentences) Answer

As you have brought out long sentences will be a problem for self-attention models: https://github.com/tensorflow/tensor2tensor/issues/843 then my questions are:

  1. Would it be feasible to achieve context with only the transformer model?
  2. If yes, then how long are too long sentences for the transformer model?
tiru1930 commented 6 years ago

@JohannesTK I am also working on the same, Did you find any way to do this kind of model?

I am able to build simple NMT model ,but i am not able to proceed further to send context to the model in case of multi turn dialogues.