When avg_ckpts is True, there are two folders to store the best model w.r.t to "best bleu" and "avg_best_bleu".
The problem is that the checkpoints file in each folder will be updated based on the file from another folder. For example,
this is the checkpoints file in folder "best_bleu" folder.
After the weights averaging, the checkpoints file in "avg_best_bleu" folder will be updated based on above file, not the file in its own. Thus, the resulting file is
However, whether to update the checkpoints is based on the comparison of its own previous metric. In this logic, it is not guaranteed the best 5 models in the checkpoints are saved. If there are more than one metric to evaluate, like bleu and rouge, it will be more messed up.
One workaround is to define multiple savers, when we have multiple metrics.
When avg_ckpts is True, there are two folders to store the best model w.r.t to "best bleu" and "avg_best_bleu".
The problem is that the checkpoints file in each folder will be updated based on the file from another folder. For example, this is the checkpoints file in folder "best_bleu" folder.
After the weights averaging, the checkpoints file in "avg_best_bleu" folder will be updated based on above file, not the file in its own. Thus, the resulting file is
However, whether to update the checkpoints is based on the comparison of its own previous metric. In this logic, it is not guaranteed the best 5 models in the checkpoints are saved. If there are more than one metric to evaluate, like bleu and rouge, it will be more messed up.
One workaround is to define multiple savers, when we have multiple metrics.