ArdalanM / nlp-benchmarks

129 stars 24 forks source link

nlp-benchmark

Datasets:

Dataset Classes Train samples Test samples source
Imdb 2 25 000 25 000 link
AG’s News 4 120 000 7 600 link
Sogou News 5 450 000 60 000 link
DBPedia 14 560 000 70 000 link
Yelp Review Polarity 2 560 000 38 000 link
Yelp Review Full 5 650 000 50 000 link
Yahoo! Answers 10 1 400 000 60 000 link
Amazon Review Full 5 3 000 000 650 000 link
Amazon Review Polarity 2 3 600 000 400 000 link

Models:

HAN word (red) and sentence (blue) attention weight at prediction:

Experiments:

Results are reported as follows: (i) / (ii)

Imdb

Model paper accuracy repo accuracy
CNN small
VDCNN 9 layers
VDCNN 17 layers
VDCNN 29 layers
HAN 90.5
Transformer 88.6

Ag news

Model paper accuracy repo accuracy
CNN small 84.35 88.30
VDCNN 9 layers 90.17 89.22
VDCNN 17 layers 90.61 90.00
VDCNN 29 layers 91.27 90.43
HAN 92.4
Transformer 93.2

Sogu news

Model paper accuracy repo accuracy
CNN small 91.35 93.53
VDCNN 9 layers 96.42 93.50
VDCNN 17 layers 96.49
VDCNN 29 layers 96.64 87.90
HAN 96.
Transformer 95.6

DBpedia

Model paper accuracy repo accuracy
CNN small 98.02 98.15
VDCNN 9 layers 98.75 98.35
VDCNN 17 layers 98.02 98.15
VDCNN 29 layers 98.71
HAN 99.0
Transformer 98.7

Yelp polarity

Model paper accuracy repo accuracy
CNN small
VDCNN 9 layers 94.73 93.97
VDCNN 17 layers 94.95 94.73
VDCNN 29 layers 95.72 94.75
HAN

Yelp review

Model paper accuracy repo accuracy
CNN small
VDCNN 9 layers 61.96 61.18
VDCNN 17 layers 62.59
VDCNN 29 layers 64.26 62.73
HAN 63.