search

minimaxir / aitextgen

A robust Python tool for text-based AI training and generation using GPT-2.
https://docs.aitextgen.io
MIT License
1.84k stars 220 forks source link

Can I finetune custom trained model? #173

Open varna9000 opened 2 years ago

varna9000 commented 2 years ago

I have a dataset of ~2GB. I can't fit it in my GPU, so I decided to split the dataset into chunks of 500MB If I train gpt-2 from scratch on first chunk of the dataset, can I use the already trained custom model for fine-tuning on the rest of the dataset chunks?

I appreciate the above problem might not be specific to aitextgen, but I can't find another workaround for fitting the big dataset into GPU.

tientr commented 2 years ago

You can, however GPT intend to bias toward to last chunk of your dataset. To avoid that, you can use merge_datasets().