Closed ilyakar closed 2 years ago
you need to have much more data, as 2 sequences is not enough. 10s of megabytes is the around the minimum required
500 lines works well with helloforefront.com (They fine-tune using GPT-J on their side), and it also works very well with OpenAI. So I don't see why 10s of MB are needed here.
Hi, Firstly, thank you so much for looking at this post. I could really use some help.
I'm trying to fine-tune GPT-J with a small dataset of ~500 lines:
Using the create_finetune_tfrecords.py script outputs a file with 2 in it. I understand that means 2 sequences.
I could really use some advice with the .json config file. What constants do you recommend for this small dataset? The best I came up with trying to follow the guide:
Very much looking forward to hearing from you! :)