Any analysis on the impact on accuracy

[qizzzh]

Just curious if any analysis done on accuracy impact

[Viol2000]

We do not change output distribution. Theoretically, the output results should be literally the same as the huggingface's greedy search outputs. In empirical experiments, sometimes results differ from huggingface's greedy search when using FP16. We owe it to floating point errors, and the accuracy is not dropped. When we use FP32, the output is exactly the same as huggingface's greedy search.

[qizzzh]

Thank you that makes sense.