Transformer - wrong BLEU evaluation during first step with Horovod

[borisgin]

When we train with --train_eval mode, the BLEU evaluation is wrong. Looks like we copy Traget into prediction? * ***EVAL Source[0]: It ' s ▁mostly ▁in ▁the ▁English - speaking ▁world . * ***EVAL Target[0]: Dies ▁ist ▁hauptsächlich ▁in ▁der ▁englisch sprechen den ▁Welt ▁der ▁Fall . * ***EVAL Prediction[0]: ounder Dies ▁ist ▁hauptsächlich ▁in ▁der ▁englisch sprechen den ▁Welt ▁der ▁Fall . ounder ounder ounder ounder ounder ounder ounder ounder ounder ounder ounder ounder ounder ounder.... Validation loss: 9.9562 Eval BLUE score: 0.962 Global step 0: Train loss: 9.7835