search

HEmile / iswc-challenge

MIT License
10 stars 1 forks source link

Is fact checking worth it? #24

Open HEmile opened 2 years ago

HEmile commented 2 years ago

Before fact checking: ChemicalCompoundElement 0.482 0.457 0.465 CompanyParentOrganization 0.745 0.760 0.748 CountryBordersWithCountry 0.631 0.625 0.619 CountryOfficialLanguage 0.623 0.627 0.592 PersonCauseOfDeath 0.500 0.500 0.500 PersonEmployer 0.276 0.335 0.270 PersonInstrument 0.568 0.552 0.531 PersonLanguage 0.755 0.936 0.797 PersonPlaceOfDeath 0.520 0.520 0.520 PersonProfession 0.719 0.514 0.567 RiverBasinsCountry 0.754 0.784 0.757 StateSharesBorderState 0.301 0.232 0.257 Average 0.573 0.570 0.552

After fact checking: ChemicalCompoundElement 0.430 0.210 0.272 CompanyParentOrganization 0.760 0.760 0.760 CountryBordersWithCountry 0.623 0.588 0.596 CountryOfficialLanguage 0.670 0.597 0.593 PersonCauseOfDeath 0.500 0.500 0.500 PersonEmployer 0.323 0.268 0.271 PersonInstrument 0.583 0.525 0.540 PersonLanguage 0.863 0.849 0.825 PersonPlaceOfDeath 0.500 0.500 0.500 PersonProfession 0.752 0.423 0.505 RiverBasinsCountry 0.763 0.733 0.735 StateSharesBorderState 0.305 0.227 0.253 Average 0.589 0.515 0.529

Seems mostly to be worse.

HEmile commented 2 years ago

Before fact checking (proper evaluation) p r f1 ChemicalCompoundElement 0.902 0.891 0.890 CompanyParentOrganization 0.485 0.500 0.488 CountryBordersWithCountry 0.830 0.794 0.792 CountryOfficialLanguage 0.834 0.843 0.793 PersonCauseOfDeath 0.560 0.560 0.560 PersonEmployer 0.266 0.315 0.256 PersonInstrument 0.589 0.570 0.551 PersonLanguage 0.762 0.936 0.801 PersonPlaceOfDeath 0.820 0.820 0.820 PersonProfession 0.720 0.513 0.569 RiverBasinsCountry 0.824 0.851 0.820 StateSharesBorderState 0.636 0.474 0.532 Average 0.686 0.672 0.656

After fact check: p r f1 ChemicalCompoundElement 0.833 0.436 0.552 CompanyParentOrganization 0.640 0.640 0.640 CountryBordersWithCountry 0.817 0.757 0.765 CountryOfficialLanguage 0.876 0.813 0.799 PersonCauseOfDeath 0.580 0.580 0.580 PersonEmployer 0.303 0.248 0.251 PersonInstrument 0.653 0.592 0.604 PersonLanguage 0.870 0.849 0.828 PersonPlaceOfDeath 0.780 0.780 0.780 PersonProfession 0.747 0.420 0.503 RiverBasinsCountry 0.843 0.799 0.805 StateSharesBorderState 0.650 0.465 0.528 Average 0.716 0.615 0.636

DimitrisAlivas commented 2 years ago

Should we disregard the first 2 results cause they were using the "wrong" evaluation?

HEmile commented 2 years ago

When only applying fact checking to relations that improve F1 p r f1 ChemicalCompoundElement 0.902 0.891 0.890 CompanyParentOrganization 0.640 0.640 0.640 CountryBordersWithCountry 0.830 0.794 0.792 CountryOfficialLanguage 0.876 0.810 0.794 PersonCauseOfDeath 0.580 0.580 0.580 PersonEmployer 0.270 0.333 0.266 PersonInstrument 0.663 0.612 0.617 PersonLanguage 0.863 0.849 0.825 PersonPlaceOfDeath 0.820 0.820 0.820 PersonProfession 0.725 0.520 0.574 RiverBasinsCountry 0.824 0.851 0.820 StateSharesBorderState 0.622 0.464 0.521 Average 0.718 0.680 0.678

(And yes Dimitris)