Bleu scores are above 1.0

[yamrgit]

I tested on the IWSLT'15 English-Vietnamese data to start learning how to use NMT.

Everything worked fine with a few minor tweaks*, but I am seeing BLEU scores above 1.0 both in training and testing,

Are you using a specific definition of BLEU that I am not aware of, or did I miss something else?

Example output from training:

b'\tstep 11500 lr 1 step-time 0.23s wps 23.68K ppl 35.14 gN 3.22 bleu 5.89, Tue Jun 12 15:05:05 2018' b'\tstep 11600 lr 1 step-time 0.17s wps 32.61K ppl 34.39 gN 3.17 bleu 5.89, Tue Jun 12 15:05:23 2018' b'\tstep 11700 lr 1 step-time 0.17s wps 33.26K ppl 34.75 gN 3.14 bleu 5.89, Tue Jun 12 15:05:40 2018' b'\tstep 11800 lr 1 step-time 0.17s wps 33.13K ppl 34.20 gN 3.06 bleu 5.89, Tue Jun 12 15:05:57 2018' b'\tstep 11900 lr 1 step-time 0.17s wps 33.21K ppl 34.71 gN 3.12 bleu 5.89, Tue Jun 12 15:06:14 2018' b'\tstep 12000 lr 1 step-time 0.17s wps 32.61K ppl 34.82 gN 3.16 bleu 5.89, Tue Jun 12 15:06:31 2018'

Example output from testing: b'\tbleu test: 4.7' b'\tsaving hparams to tmp/nmt_model\hparams'

Thanks.

*on Windows, the following lines of code in misc_utils.py do not work (L63-67): out_s = s.encode("utf-8") if not isinstance(out_s, str): out_s = out_s.decode("utf-8") print(out_s, end="", file=sys.stdout)

I changed them to: if isinstance(out_s, str): out_s = out_s.encode("utf-8") print(out_s, end="", file=sys.stdout)

which solved the issue, and seemed more portable

** the notation "from .packagename import" and "from ..packagename import" seems to be IDE specific

[mommi84]

The BLEU score is expressed in percentage, so it means 5.89% or 0.0589.

[hpulfc]

@ymrabet maybe you should put the hparams !

[yamrgit]

Thanks @mommi84 ! I wasn't expecting it to be that low, so maybe @hpulfc is right, I just ran it as is for that first test.