PaddlePaddle / PaddleNLP

👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis etc.

https://paddlenlp.readthedocs.io

Apache License 2.0

12.08k stars 2.93k forks source link

【模型更新】ELECTRA-180g和ELECTRA-ex #463

Closed ghost closed 2 years ago

ghost commented 3 years ago

希望能增加ELECTRA-180g和ELECTRA-*-ex两个模型的预训练参数。

模型	开发集	测试集	挑战集	参数量
BERT-base	65.5 (64.4) / 84.5 (84.0)	70.0 (68.7) / 87.0 (86.3)	18.6 (17.0) / 43.3 (41.3)	102M
BERT-wwm	66.3 (65.0) / 85.6 (84.7)	70.5 (69.1) / 87.4 (86.7)	21.0 (19.3) / 47.0 (43.9)	102M
BERT-wwm-ext	67.1 (65.6) / 85.7 (85.0)	71.4 (70.0) / 87.7 (87.0)	24.0 (20.0) / 47.3 (44.6)	102M
RoBERTa-wwm-ext	67.4 (66.5) / 87.2 (86.5)	72.6 (71.4) / 89.4 (88.8)	26.2 (24.6) / 51.0 (49.1)	102M
RBT3	57.0 / 79.0	62.2 / 81.8	14.7 / 36.2	38M
**ELECTRA-small	63.4 (62.9) / 80.8 (80.2)	67.8 (67.4) / 83.4 (83.0)	16.3 (15.4) / 37.2 (35.8)	12M
ELECTRA-180g-small	63.8 / 82.7	68.5 / 85.2	15.1 / 35.8	12M
ELECTRA-small-ex	66.4 / 82.2	71.3 / 85.3	18.1 / 38.3	25M
ELECTRA-180g-small-ex	68.1 / 85.1	71.8 / 87.2	20.6 / 41.7	25M
ELECTRA-base	68.4 (68.0) / 84.8 (84.6)	73.1 (72.7) / 87.1 (86.9)	22.6 (21.7) / 45.0 (43.8)	102M
ELECTRA-180g-base	69.3 / 87.0	73.1 / 88.6	24.0 / 48.6	102M
ELECTRA-large	69.1 / 85.2	73.9 / 87.1	23.0 / 44.2	324M
ELECTRA-180g-large	68.5 / 86.2	73.5 / 88.5	21.8 / 42.9	324M**

ZeyuChen commented 3 years ago

@illcat 请问你想要这些模型做哪些问题？也欢迎加入我们特殊兴趣小组SIG，一起参与模型复现 :) 可否加下相关联系方式？

ghost commented 3 years ago

https://github.com/ymcui/Chinese-ELECTRA 提供了ECELTRA的最新进展与TF权重。

已申请加入SIG-QQ讨论群，希望通过下

ZeyuChen commented 3 years ago

留下你的QQ号码或者微信号？我单独加你直接拉群 @illcat

yingyibiao commented 2 years ago

1086