Closed peter-ni-noob closed 3 weeks ago
Yes, he elaborated on this topic in his video. Overfitting is not a major concern for this project overall because the dataset is virtually infinite. Even after 4 epochs and 40 billion tokens, the validation loss is decreasing steadily.
ok
peter-ni-noob
commented
3 weeks ago
Yes, he elaborated on this topic in his video. Overfitting is not a major concern for this project overall because the dataset is virtually infinite. Even after 4 epochs and 40 billion tokens, the validation loss is decreasing steadily.