search

DRAGNLabs / 301r_retnet

2 stars 1 forks source link

Split data random #58

Open calvin-seamons opened 3 months ago

calvin-seamons commented 3 months ago

So basically this branch allows the user to specify a certain percentage of C4 to train the model on. It is somewhat outlined in the template_config.yaml file.

If you have any questions let me know. I may have included some unnecessary changes but overall it should be good. Just trained a 70M model on 1% of C4 so it works at least for me

nprisbrey commented 3 months ago

This is a quick review getting the clear issues I see. Please resolve them and then I will look more carefully at train_model.py itself.

@calvin-seamons, this is a good review by @KimballNJardine. I'll take another look at it if desired after these changes have been addressed.

DrewGalbraith commented 3 months ago

You did a lot of good work in this PR, I think it will be great to have this around to run small models.

calvin-seamons commented 3 months ago

Testing it rn when the queue is fixed

DrewGalbraith commented 2 months ago

@KimballJardine @nprisbrey, where are we at on this?