search

keyu-tian / SparK

[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"
https://arxiv.org/abs/2301.03580
MIT License
1.42k stars 82 forks source link

q #7

Closed C929-x closed 1 year ago

C929-x commented 1 year ago

Hello, I don't know if you has any research involving NLP, I want to ask if there is any better pre-training model now, or how to improve Bert

keyu-tian commented 1 year ago

For NLP there are some subsequent work of BERT like RoBERTa/DeBERTa/Electra, and I would recommend a try.