search

Wikidata / soweego

Link Wikidata items to large catalogs
https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego_2
GNU General Public License v3.0
95 stars 8 forks source link

Analyze performance of ensemble methods #359

Closed tupini07 closed 4 years ago

tupini07 commented 4 years ago

Analyze the performance that the new ensemble methods methods have for linking and evaluation, and report said results here.

Evaluation:

Linking:

tupini07 commented 4 years ago

Evaluation Performance

Catalog Entity Model F1.Mean F1.STD Prec.Mean Prec.STD Recall.Mean Recall.STD
discogs musician gated_classifier 0.936638 0.002652 0.922472 0.004920 0.951308 0.006874
discogs band gated_classifier 0.927455 0.001522 0.912845 0.002324 0.942566 0.004981
imdb director gated_classifier 0.918505 0.001411 0.900678 0.002997 0.937067 0.003067
imdb musician gated_classifier 0.924625 0.002018 0.943199 0.010644 0.906929 0.007107
imdb producer gated_classifier 0.910555 0.003934 0.902453 0.007668 0.918955 0.010547
imdb writer gated_classifier 0.931719 0.002789 0.928561 0.007822 0.934984 0.005804
imdb actor gated_classifier 0.897257 0.002191 0.895105 0.003315 0.899486 0.007650
musicbrainz band gated_classifier 0.918238 0.003816 0.948221 0.012288 0.890520 0.016863
musicbrainz musician gated_classifier 0.955526 0.001926 0.951364 0.006050 0.959821 0.007915
discogs band stacked_classifier 0.927771 0.001080 0.911809 0.006866 0.944456 0.009776
discogs musician stacked_classifier 0.938836 0.002473 0.917001 0.002568 0.961746 0.003667
imdb musician stacked_classifier 0.924930 0.001915 0.947724 0.003561 0.903238 0.004935
imdb actor stacked_classifier 0.890012 0.007978 0.907933 0.016179 0.873941 0.029767
imdb producer stacked_classifier 0.904998 0.008882 0.899323 0.020682 0.912508 0.035943
imdb writer stacked_classifier 0.931043 0.003075 0.929588 0.007946 0.932652 0.009692
imdb director stacked_classifier 0.917923 0.001777 0.901563 0.004736 0.934969 0.007596
musicbrainz musician stacked_classifier 0.955643 0.001373 0.955374 0.003750 0.955933 0.003327
musicbrainz band stacked_classifier 0.919791 0.003254 0.941948 0.011043 0.898935 0.013037
discogs band voting_classifier_hard 0.926795 0.001145 0.905189 0.001827 0.949465 0.002661
discogs musician voting_classifier_hard 0.939658 0.001932 0.913881 0.002899 0.966938 0.002004
imdb director voting_classifier_hard 0.918177 0.001400 0.890278 0.004667 0.947915 0.003379
imdb producer voting_classifier_hard 0.907517 0.004611 0.901155 0.004867 0.914052 0.009619
imdb actor voting_classifier_hard 0.897184 0.000752 0.882077 0.009962 0.913098 0.012148
imdb writer voting_classifier_hard 0.931065 0.002249 0.920422 0.007199 0.942038 0.005549
imdb musician voting_classifier_hard 0.921577 0.002711 0.918733 0.022289 0.925297 0.017423
musicbrainz musician voting_classifier_hard 0.956088 0.001293 0.946047 0.003334 0.966356 0.001537
musicbrainz band voting_classifier_hard 0.917456 0.002139 0.953082 0.007505 0.884490 0.006698
discogs band voting_classifier_soft 0.927576 0.001300 0.904116 0.002522 0.952312 0.004592
discogs musician voting_classifier_soft 0.938896 0.001494 0.909274 0.004537 0.970543 0.003097
imdb actor voting_classifier_soft 0.899699 0.001028 0.887665 0.005118 0.912139 0.006529
imdb director voting_classifier_soft 0.919262 0.001497 0.894902 0.008123 0.945108 0.006606
imdb producer voting_classifier_soft 0.909610 0.003182 0.905090 0.005243 0.914192 0.002709
imdb musician voting_classifier_soft 0.926019 0.001244 0.939099 0.007632 0.913418 0.007684
imdb writer voting_classifier_soft 0.931617 0.002161 0.929887 0.004030 0.933369 0.002867
musicbrainz musician voting_classifier_soft 0.954936 0.001831 0.955208 0.002438 0.954666 0.001676
musicbrainz band voting_classifier_soft 0.916297 0.002493 0.954029 0.002458 0.881462 0.005454

Metrics Summary

Sorted by F1 (descending)

Model Average F1 Average F1.STD Average Prec Average Prec.STD Average Recall Average Recall.STD
voting_classifier_soft 0.924879 0.001803 0.919919 0.004678 0.930801 0.004579
gated_classifier 0.924502 0.002473 0.922766 0.006448 0.926848 0.007868
voting_classifier_hard 0.923946 0.002026 0.914540 0.007172 0.934405 0.006780
stacked_classifier 0.923439 0.003534 0.923585 0.008592 0.924264 0.013082

Graphics

F1 Scores

ens_f1

Precision Scores

ens_prec

Recall Scores

ens_rec

Precision vs Recall

ens_precvfrec

tupini07 commented 4 years ago

For Linking

Continuous classifiers:

ens_classif_contin

Binary classifiers:

ens_classif_binary

Catalog Entity Model Count Mean STD Min 25% 50% 75% Max
discogs musician gated_classifier 515082 0.029858 0.134515 0.000000 0.005109 0.005152 0.005247 0.984798
discogs band gated_classifier 92137 0.057388 0.179030 0.000000 0.009710 0.009955 0.010281 0.981458
imdb actor gated_classifier 355515 0.065065 0.181855 0.005060 0.005130 0.005192 0.005239 0.987478
imdb musician gated_classifier 944196 0.017756 0.079748 0.004216 0.004266 0.004288 0.004328 0.988251
imdb writer gated_classifier 68702 0.048981 0.170615 0.005432 0.005505 0.005550 0.005590 0.989302
imdb producer gated_classifier 9351 0.080576 0.213030 0.006867 0.007220 0.007352 0.007708 0.993112
imdb director gated_classifier 35094 0.066865 0.202458 0.005077 0.005161 0.005196 0.005438 0.983906
musicbrainz musician gated_classifier 652175 0.027414 0.123071 0.000000 0.006402 0.006426 0.006457 1.000000
musicbrainz band gated_classifier 106026 0.082290 0.218746 0.000000 0.013210 0.015158 0.018073 1.000000
discogs band stacked_classifier 92137 0.054855 0.171827 0.000000 0.012356 0.012482 0.013273 0.959502
discogs musician stacked_classifier 515082 0.023429 0.111203 0.000000 0.004903 0.004904 0.004925 0.956270
imdb actor stacked_classifier 355515 0.061541 0.175259 0.006668 0.006805 0.006814 0.007155 0.982776
imdb producer stacked_classifier 9351 0.106723 0.253498 0.014097 0.014177 0.015255 0.018345 0.970012
imdb musician stacked_classifier 944196 0.018207 0.082833 0.005190 0.005191 0.005253 0.005348 0.974544
imdb director stacked_classifier 35094 0.061921 0.188322 0.005990 0.006000 0.006003 0.006119 0.956035
imdb writer stacked_classifier 68702 0.056798 0.194364 0.006776 0.006781 0.006801 0.007176 0.972416
musicbrainz musician stacked_classifier 652175 0.027208 0.115886 0.000000 0.009073 0.009075 0.009164 1.000000
musicbrainz band stacked_classifier 106026 0.093695 0.222461 0.000000 0.018945 0.020480 0.035917 1.000000
discogs band voting_classifier_hard 92137 0.057187 0.232200 0.000000 0.000000 0.000000 0.000000 1.000000
discogs musician voting_classifier_hard 515082 0.029859 0.170199 0.000000 0.000000 0.000000 0.000000 1.000000
imdb musician voting_classifier_hard 337111 0.017525 0.131219 0.000000 0.000000 0.000000 0.000000 1.000000
imdb writer voting_classifier_hard 68702 0.048718 0.215279 0.000000 0.000000 0.000000 0.000000 1.000000
imdb actor voting_classifier_hard 355515 0.083248 0.276257 0.000000 0.000000 0.000000 0.000000 1.000000
imdb producer voting_classifier_hard 9351 0.087156 0.282079 0.000000 0.000000 0.000000 0.000000 1.000000
imdb director voting_classifier_hard 35094 0.081125 0.273031 0.000000 0.000000 0.000000 0.000000 1.000000
musicbrainz band voting_classifier_hard 106026 0.061240 0.239771 0.000000 0.000000 0.000000 0.000000 1.000000
musicbrainz musician voting_classifier_hard 652175 0.022537 0.148422 0.000000 0.000000 0.000000 0.000000 1.000000
discogs band voting_classifier_soft 92137 0.063509 0.168745 0.000000 0.008184 0.012160 0.017549 0.999697
discogs musician voting_classifier_soft 515082 0.030028 0.133205 0.000000 0.001535 0.002331 0.004532 0.998474
imdb musician voting_classifier_soft 944196 0.021848 0.092686 0.000205 0.001234 0.001659 0.002589 0.999709
imdb writer voting_classifier_soft 68702 0.043952 0.148484 0.000438 0.002171 0.002948 0.003702 0.999094
imdb producer voting_classifier_soft 9351 0.076109 0.182346 0.002841 0.006002 0.006845 0.009239 0.996601
imdb actor voting_classifier_soft 355515 0.068285 0.185416 0.000203 0.002040 0.002958 0.004662 0.998976
imdb director voting_classifier_soft 35094 0.064267 0.185011 0.000541 0.002337 0.002876 0.007271 0.999539
musicbrainz band voting_classifier_soft 106026 0.126356 0.213054 0.000000 0.004267 0.034199 0.137530 1.000000
musicbrainz musician voting_classifier_soft 652175 0.024370 0.115483 0.000000 0.001089 0.001672 0.002279 1.000000

Below is a summary of the averaged values for each kind of ensemble

Model Average Mean Average STD Average 25% Average 50% Average 75% Average Max
voting_classifier_soft 0.057636 0.158270 0.003207 0.007516 0.021039 0.999121
stacked_classifier 0.056042 0.168406 0.009359 0.009674 0.011936 0.974617
voting_classifier_hard 0.054288 0.218717 0.000000 0.000000 0.000000 1.000000
gated_classifier 0.052910 0.167008 0.006857 0.007141 0.007596 0.989812