search

venkatasg / Lil-Bevo

UT Austin's submission to BabyLM Challenge
https://huggingface.co/collections/venkatasg/babylm-653591cdb66f4bf68922873a
MIT License
2 stars 2 forks source link

Contrastive loss #9

Closed venkatasg closed 1 year ago

venkatasg commented 1 year ago

Need to setup BevoEncoder for MLM, or use off the shelf encoder.

[CLS] good . [CLS] prompt > [CLS]. bad [CLS].prompt

Per token basis like Electra?

What corruptions are allowed? As long we use the tokens from the data is anything fine?

venkatasg commented 1 year ago

Didn't really work?

juand-r commented 1 year ago

We didn't really fully explore the space...