Applying F.log_softmax on Generator otuput

[robinsongh381]

Hi

You have applied F.log_sofmax on the output of projection layer in [line 232] (https://github.com/HLTCHKUST/PAML/blob/3c1fe4e55956b74fe0682b431726e5396f8db490/model/transformer.py#L232).

If we use nn.CrossEntropy for the loss function, the result of F.log_softmax enters in the loss function as in [line 333] (https://github.com/HLTCHKUST/PAML/blob/3c1fe4e55956b74fe0682b431726e5396f8db490/model/transformer.py#L333)

So basically the output of the projection layer goes through F.log_softmax and then nn.CrossEntropy.

However, if you look at here, simply applying nn.CrossEntropy would automatically apply F.log_softmax so I think you should exlude the line 232 and instead just return [line 218] (https://github.com/HLTCHKUST/PAML/blob/3c1fe4e55956b74fe0682b431726e5396f8db490/model/transformer.py#L218).

What do you think ?

[zlinao]

indeed we use self.criterion = nn.NLLLoss(ignore_index=config.PAD_idx) instead of nn.CrossEntropy

[zlinao]

you can also use logits + CrossEntropy