search

PiotrNawrot / nanoT5

Fast & Simple repository for pre-training and fine-tuning T5-style models
Apache License 2.0
970 stars 74 forks source link

Pre-train on different Dataset than C4 #27

Closed nikifori closed 9 months ago

nikifori commented 10 months ago

Is there any option, to pre-train T5-based models on different dataset than C4 in a self-supervised manner?

PiotrNawrot commented 9 months ago

@nikifori

Please accept my apologies for the late reply. I've been very busy lately because of the ICML submission.

To answer your question: Take a look on : https://github.com/PiotrNawrot/nanoT5/issues/11 and https://github.com/PiotrNawrot/nanoT5/issues/21

Training on a different dataset seems like a common question so I'll try to add some more information about it to the README.

I'm closing this Issue for now, please feel free to re-open if you have any other questions.