Stuff we want to try for loose track

venkatasg / Lil-Bevo

UT Austin's submission to BabyLM Challenge

MIT License

2 stars 2 forks source link

Closed venkatasg closed 1 year ago

venkatasg commented 1 year ago

Stuff we want to try for loose track.

[ ] Can we use models trained on a larger amount of data to identify training dataset?
[ ] RLHF w/ GPT4 output
[ ] Generate texts from GPT4 based on the current model’s predictions.
[ ] Symbolic distillation
[ ] cheat and put BLIMP / SuperGLUE stuff there

venkatasg commented 1 year ago

Didn't try.