search

Joshua-Ren / Neural_Iterated_Learning

Pytorch implementation of the paper 'Compositional language emerge in a neural iterated learning' (ICLR 2020).
MIT License
13 stars 2 forks source link

Potential bug on listener logprobs #2

Open JACKHAHA363 opened 3 years ago

JACKHAHA363 commented 3 years ago

https://github.com/Joshua-Ren/Neural_Iterated_Learning/blob/master/train.py#L93

In this line, the listener logprobs is computed on the batch axis, which does not seem correct?

The correct one would be: F.log_softmax(pred_vector, dim=1).max(dim=1)[0]

Joshua-Ren commented 3 years ago

Hi,

Thanks very much for your email. And thanks for pointing out this bug in the code. Actually, I also noticed this bug recently. But strangely, I found this loss works so well. I have tried some different versions, including the correct implementation (i.e., Softmax(pred_vector)[true_id].log().mean()), the cross-entropy loss between logits and predictions, etc. But this loss performs best. Anyway, it is also safe to use cross-entropy loss instead, which seems more reasonable.

Best,

Joshua

Yuchen Lu notifications@github.com 于2021年2月9日周二 上午5:11写道：

https://github.com/Joshua-Ren/Neural_Iterated_Learning/blob/master/train.py#L93

In this line, the listener logprobs is computed on the batch axis, which does not seem correct?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Joshua-Ren/Neural_Iterated_Learning/issues/2, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGGLYAJ7VX7QGC3QG4EMR4DS6BHOLANCNFSM4XJXSJFQ .