search

neccam / nslt

Neural Sign Language Translation (CVPR'18)
Apache License 2.0
149 stars 41 forks source link

Bleu scores #7

Open iliasprc opened 5 years ago

iliasprc commented 5 years ago

When i run the inference with the trained model with the evaluation ,the train and the test set i get the following BLEU-4 scores Train set: 2.0 Dev set: 8.4 Test set: 9.3 How is that possible to get lower scores using the training set?

neccam commented 5 years ago

Hi,

This is quite interesting and have never encountered myself. Would you mind sharing the commands you have used?

iliasprc commented 5 years ago

I have used the following command for the inference model python -m nmt --out_dir=trained_model --inference_input_file=Data/phoenix2014T.test.sign --inference_output_file=Data/predictions_test.de --vocab_prefix= Data/phoenix2014T.vocab --inference_ref_file=Data/phoenix2014T.test.de --base_gpu=0

where every time i change the inference input and ref files to train, test , dev

neccam commented 5 years ago

Thanks a lot, I will investigate to see what went wrong with it.

MeteorsHub commented 5 years ago

Same issue. The scores on my machine are Train set: 1.8 Dev set: 9.3 Test set: 8.5