search

arianhosseini / MemArchs-in-RNNLM

attempt at implementing "Memory Architectures in Recurrent Neural Network Language Models" as a part of the ICLR 2018 reproducibility challenge
5 stars 2 forks source link

different implementation from original paper for policy network #2

Open marcwww opened 5 years ago

marcwww commented 5 years ago

Dear author, I've found that in your code, the policy is estimated by a conv1d over the memory followed by a linear transformation, which is different from the recursive formulations in the paper(showing below:) a If you could do some explanation for me, that will be really appreciated.

jpilaul commented 5 years ago

I just sent the authors of the original paper Memory architectures in Recurrent Neural Network Language Models a note about this because I was not clear to me either how the recursion was done and how the probabilities were calculated. I would have used an MLP but perhaps Dani Yogatama used something different. This part is not transparent in the original paper... I found some of the authors of the original paper on github (I couldn't find Dani Yogatam): @ysmiao, @melisgl, @wlin12, @adhigunasurya can perhaps help shed some light on the matter.

jpilaul commented 5 years ago

Dani Yogatama, the main author, has answered my email. The recursion of the adaptive stack rnn makes total sense now. The first observation that he made was that it should an MLP not a convolutional layer. Second, Dani detailed the equations of the recursion in the email. I will create a pull request with the updates. Thank you to Dani and all the authors of Memory architectures in Recurrent Neural Network for quick answers on this.

marcwww commented 5 years ago

Dani Yogatama, the main author, has answered my email. The recursion of the adaptive stack rnn makes total sense now. The first observation that he made was that it should an MLP not a convolutional layer. Second, Dani detailed the equations of the recursion in the email. I will create a pull request with the updates. Thank you to Dani and all the authors of Memory architectures in Recurrent Neural Network for quick answers on this.

Hey jpilaul, what u have done is really cheerful. Could u please forward the email to me? My address is marcwang@mail.ru Thanks a lot.