search

aayush97 / semeval2023-afrisenti

A low-resource sentiment analysis project for African Languages
MIT License
0 stars 0 forks source link

Evaluate a LinearSVC and naive bayes baseline on the african dataset directly (without translations) #5

Closed aayush97 closed 1 year ago

aayush97 commented 1 year ago

Use pretrained tokenizer from huggingface 'Davlan/afro-xlmr-mini'

aayush97 commented 1 year ago

Linear SVC results

  model language num_examples precision recall f1_score_macro accuracy
0 Linear SVC am 5984 {'NEGATIVE': 0.5171339563862928, 'POSITIVE': 0.6998444790046656, 'NEUTRAL': 0.5752287720791658} {'NEGATIVE': 0.2144702842377261, 'POSITIVE': 0.33783783783783783, 'NEUTRAL': 0.8708118556701031} 0.483901002359983 0.582386363636364
1 Linear SVC dz 1651 {'NEGATIVE': 0.5930735930735931, 'POSITIVE': 0.6042780748663101, 'NEUTRAL': 0.47435897435897434} {'NEGATIVE': 0.92152466367713, 'POSITIVE': 0.2709832134292566, 'NEUTRAL': 0.10818713450292397} 0.424016116940664 0.588734100545124
2 Linear SVC ha 14172 {'NEGATIVE': 0.72371934935664, 'POSITIVE': 0.8596810933940775, 'NEUTRAL': 0.665371711107187} {'NEGATIVE': 0.6518696698010059, 'POSITIVE': 0.8052058886281204, 'NEUTRAL': 0.7671009771986971} 0.743364824683949 0.742520462884561
3 Linear SVC ig 10192 {'NEGATIVE': 0.7888138862102217, 'POSITIVE': 0.864023712486106, 'NEUTRAL': 0.7263332718213693} {'NEGATIVE': 0.6292307692307693, 'POSITIVE': 0.7561608300907912, 'NEUTRAL': 0.8731144631765749} 0.766511141314108 0.775510204081633
4 Linear SVC ma 5583 {'NEGATIVE': 0.7545180722891566, 'POSITIVE': 0.7960264900662252, 'NEUTRAL': 0.7256830601092896} {'NEGATIVE': 0.6021634615384616, 'POSITIVE': 0.6837315130830489, 'NEUTRAL': 0.9217954650624711} 0.739157022740557 0.751567257746731
5 Linear SVC pcm 5121 {'NEGATIVE': 0.7253797158255757, 'POSITIVE': 0.7225433526011561, 'NEUTRAL': 0.0} {'NEGATIVE': 0.9136069114470843, 'POSITIVE': 0.41482300884955753, 'NEUTRAL': 0.0} 0.445246827230871 0.724663151728178
6 Linear SVC pt 3063 {'NEGATIVE': 0.6170212765957447, 'POSITIVE': 0.752, 'NEUTRAL': 0.602510460251046} {'NEGATIVE': 0.3337595907928389, 'POSITIVE': 0.27606461086637296, 'NEUTRAL': 0.9} 0.519622113968977 0.616715638263141
7 Linear SVC sw 1810 {'NEGATIVE': 1.0, 'POSITIVE': 0.584070796460177, 'NEUTRAL': 0.6090801886792453} {'NEGATIVE': 0.005235602094240838, 'POSITIVE': 0.1206581352833638, 'NEUTRAL': 0.9636194029850746} 0.318934649967887 0.607734806629834
8 Linear SVC yo 8522 {'NEGATIVE': 0.6843335931410756, 'POSITIVE': 0.751039501039501, 'NEUTRAL': 0.6924211147154232} {'NEGATIVE': 0.469017094017094, 'POSITIVE': 0.8159232072275551, 'NEUTRAL': 0.7554697554697555} 0.687095863324566 0.717671908002816

aayush97 commented 1 year ago
Multinomial Naive Bayes results   model language num_examples precision recall f1_score_macro accuracy
0 Multinomial Naive Bayes am 5984 {'NEGATIVE': 0.43563766388557806, 'POSITIVE': 0.55, 'NEUTRAL': 0.6159047005795235} {'NEGATIVE': 0.4722222222222222, 'POSITIVE': 0.4954954954954955, 'NEUTRAL': 0.6163015463917526} 0.530207627410275 0.552139037433155
1 Multinomial Naive Bayes dz 1651 {'NEGATIVE': 0.5973782771535581, 'POSITIVE': 0.4583333333333333, 'NEUTRAL': 0.2857142857142857} {'NEGATIVE': 0.7152466367713004, 'POSITIVE': 0.44844124700239807, 'NEUTRAL': 0.14619883040935672} 0.432592446391839 0.529981829194428
2 Multinomial Naive Bayes ha 14172 {'NEGATIVE': 0.7006688963210702, 'POSITIVE': 0.8618705035971223, 'NEUTRAL': 0.6385832187070152} {'NEGATIVE': 0.6413732779357096, 'POSITIVE': 0.7668017921911671, 'NEUTRAL': 0.7561074918566775} 0.724555455685754 0.722622071690658
3 Multinomial Naive Bayes ig 10192 {'NEGATIVE': 0.6240482233502538, 'POSITIVE': 0.826710816777042, 'NEUTRAL': 0.7540490513651087} {'NEGATIVE': 0.7565384615384615, 'POSITIVE': 0.7285992217898832, 'NEUTRAL': 0.7229370008873115} 0.732220621349017 0.733222135007849
4 Multinomial Naive Bayes ma 5583 {'NEGATIVE': 0.732728541521284, 'POSITIVE': 0.8387334315169367, 'NEUTRAL': 0.7234957020057307} {'NEGATIVE': 0.6310096153846154, 'POSITIVE': 0.6478953356086462, 'NEUTRAL': 0.934752429430819} 0.741602765966748 0.753895754970446
5 Multinomial Naive Bayes pcm 5121 {'NEGATIVE': 0.761688863700765, 'POSITIVE': 0.6442065491183879, 'NEUTRAL': 0.0} {'NEGATIVE': 0.8293736501079914, 'POSITIVE': 0.5658185840707964, 'NEUTRAL': 0.0} 0.46552169291181 0.724663151728178
6 Multinomial Naive Bayes pt 3063 {'NEGATIVE': 0.4633920296570899, 'POSITIVE': 0.6086956521739131, 'NEUTRAL': 0.6792963464140731} {'NEGATIVE': 0.639386189258312, 'POSITIVE': 0.4522760646108664, 'NEUTRAL': 0.6275} 0.569557510900463 0.591576885406464
7 Multinomial Naive Bayes sw 1810 {'NEGATIVE': 0.37373737373737376, 'POSITIVE': 0.4625228519195612, 'NEUTRAL': 0.6606529209621993} {'NEGATIVE': 0.193717277486911, 'POSITIVE': 0.4625228519195612, 'NEUTRAL': 0.7173507462686567} 0.468510228702075 0.585082872928177
8 Multinomial Naive Bayes yo 8522 {'NEGATIVE': 0.5766541151156536, 'POSITIVE': 0.7331869338457315, 'NEUTRAL': 0.6874172185430464} {'NEGATIVE': 0.5726495726495726, 'POSITIVE': 0.7540937323546019, 'NEUTRAL': 0.667953667953668} 0.665227982746224 0.682820934053039