search

WoosukKwon / retraining-free-pruning

[NeurIPS 2022] A Fast Post-Training Pruning Framework for Transformers
https://arxiv.org/abs/2204.09656
173 stars 27 forks source link

GLUE & SQuAD Metadata #2

Closed WoosukKwon closed 2 years ago

WoosukKwon commented 3 years ago

GLUE Benchmark Metadata

Task MNLI QQP QNLI SST-2 CoLA STS-B MRPC RTE
Train set size 393K 364K 105K 67K 8.5K 7K 3.7K 2.5K
Dev set size 20K 40K 5.4K 0.8K 1K 1.5K 0.4K 0.3K
Test set size 20K 391K 5.4K 1.8K 1K 1.4K 1.7K 3K
Med. sequence length (train) 38 28 48 11 24 54 57
Med. sequence length (dev) 37 28 45 25 29 54 54

SQuAD Benchmark Metadata

Task SQuAD V1.1 SQuAD 2.0

GLUE dev set Evaluation time (in sec)

Task MNLI QQP QNLI SST-2 CoLA STS-B MRPC RTE
V100 (FP32, batch 128) 24.39 76.39 14.43 1.41 2.31 1.09

SQuAD Evaluation time (in sec)

Task SQuAD V1.1 SQuAD 2.0
V100 (FP32, batch 128) 110.11 115.59