sharing weights of the pretrained stage model

[jasonppy]

Hi Zhifeng,

Since there are a few datasets that I cannot obtain, is it possible to share the weights of the model after the pretraining stage?

Also this is the training loss of pretraining stage (quite high variance):

and this is the training loss of the SFT stage

Do them look right to you?

My reproduced results are a bit far from reported results. And sharing the weights of the model after the pretraining stage will great help me narrow down the issue.

Appreciate your time and effort!

Puyuan

[zhifengkongnv]

We could not opensource the additional model checkpoint due to review policy.

The training loss (SFT) looks fine to me - if you set smoothing=0.6 then it oscillates between 0.5 and 1.5.

The validation loss is more informative. For example, the valid_losses of 3 different validation sets below indicate the model learns well on these tasks.