Performance Drop Issues (AIDA test A)

[liehe]

According to the paper, the performance PBoH on AIDA test A is 86.63/85.48. Due to the upgrade of gerbil, the performance of PBoH is give here is 75.19/73.3.

However, when try to reproduce the result, it gives the following result (64.84/64.32).

############### RESULTS for dataset AIDA test A for TEST w = loopybeliefpropagation.ScorerWeights@3cb5cdba params a = 0.5, f = 1.0, g = 0.5, h = 1.0, s = 0.0, b = 0.075 ################# Num total docs = 216 Num total mentions (including duplicates) = 4781

Looking at docs with GLOBAL mentions: GLOBAL mentions : num docs evaluated = 216; num mentions in solution = 4065.0 num mentions in ground truth = 4781.0

################################# GLOBAL mentions : micro F1 (per mention) Loopy : 64.84286683246664 GLOBAL mentions : micro accuracy/recall (per mention) Loopy : 59.987450324199955 GLOBAL mentions : MACRO F1 (per doc) Loopy : 64.3229987260122 GLOBAL mentions : MACRO accuracy/recall (per doc) Loopy : 58.679661769593 ###################################

GLOBAL mentions : micro F1 (per mention) ARGMAX : 62.85326701333936 GLOBAL mentions : micro acc/recall (per mention) ARGMAX : 58.1468312068605 GLOBAL mentions : MACRO F1 (per doc) ARGMAX : 61.799181989071386 GLOBAL mentions : MACRO acc/recall (per doc) ARGMAX : 56.37695767780255

GLOBAL mentions : MACRO (per doc) common Loopy - ARGMAX : 92.41201422346343 GLOBAL mentions : micro (per mention) common Loopy - ARGMAX : 94.98154981549816 GLOBAL mentions : micro (per mention) perc missing mentions from index : 14.975946454716587 GLOBAL mentions : micro (per mention) perc missing entities from mention index : 17.025726835390085

---

GLOBAL mentions : avg LBP running time (milliseconds) : 71.80092592592592 GLOBAL mentions : avg num iters in LBP : 2.6805555555555554 GLOBAL mentions : percentage cases where LBP converged: 100.0 GLOBAL mentions : avg num candidates per mention: 5.67029883619935

==============================================

1. I used the index file from polybox. The location are index files are updated.

2. I changed from

   val file = "/media/hofmann-scratch/Octavian/entity_linking/marinah/AIDA/testa_testb_aggregate"

to "AIDA-YAGO2-dataset.tsv" which is generated by files downloaded from MPI-info.

3. I use

   java -Xmx90g -cp target/PBoH-1.0-SNAPSHOT-jar-with-dependencies.jar el.EL_LBP_Spark testPBOHOnAllDatasets max-product

   to run the code because the command

   scala -J-Xmx90g target/PBoH-1.0-SNAPSHOT-jar-with-dependencies.jar testPBOHOnAllDatasets max-product

   will generate a UnstaisfiedLinkError when it trys to use leveldbjni.

Did I made any mistakes in the process? How can I reproduce the result in Gerbil?

Thanks.

[octavian-ganea]

The ARGMAX results represent the "Local Mention" prior and they should be much higher cf Table 3 in our paper. What p(e|m) indexes do you use ? Are you using the ones that we provided, to be found here: https://polybox.ethz.ch/index.php/s/IOWjGrU3mjyzDSV/authenticate

It seems there is a big overlap between PBOH and LocalMention (the "common Loopy - ARGMAX" part). I will try to re-run it tonight on a fresh machine if you still cannot solve this issue. Can you please send me your full output log file by e-mail ?

[liehe]

Hi, thanks for your fast response.

I have not changed the method or the index itself. All I have changed is updating the index address in code, add UTF-8 encoding when using Source.fromFile(), and the AIDA dataset name (The one given in AIDA.scala is "testa_testb_aggregate" which I didn't a file with this name so I used the output file from "aida-yago2-dataset.jar" ). Also, I only ran the AIDA test A and ignored all the other dataset to save time.

I am going run it again to see if the result is the same. If so, I will send you the output log.

[octavian-ganea]

I am not sure what is the output file from "aida-yago2-dataset.jar", but your testa_testb_aggregate should contain the AIDA-A and AIDA-B datasets and be generated as described on the MPI website. It should look as follows (sorry, it has a license from MPI and I cannot upload the full file myself). One word per each line, with annotations when the word is part of a mention, tab separated:

-DOCSTART- (947testa CRICKET)
CRICKET
-
LEICESTERSHIRE  B   LEICESTERSHIRE  Leicestershire_County_Cricket_Club  http://en.wikipedia.org/wiki/Leicestershire_County_Cricket_
Club    1622318 /m/05hf4j
TAKE
OVER
AT
TOP
AFTER
INNINGS
VICTORY
.

LONDON  B   LONDON  London  http://en.wikipedia.org/wiki/London 17867   /m/04jpl
1996-08-30

West    B   West Indian West_Indies_cricket_team    http://en.wikipedia.org/wiki/West_Indies_cricket_team   3379941 /m/098knd
Indian  I   West Indian West_Indies_cricket_team    http://en.wikipedia.org/wiki/West_Indies_cricket_team   3379941 /m/098knd
all-rounder
Phil    B   Phil Simmons    Phil_Simmons    http://en.wikipedia.org/wiki/Phil_Simmons   2518836 /m/07kgj4
Simmons I   Phil Simmons    Phil_Simmons    http://en.wikipedia.org/wiki/Phil_Simmons   2518836 /m/07kgj4
took
four
for
38
on
Friday
as
Leicestershire  B   Leicestershire  Leicestershire_County_Cricket_Club  http://en.wikipedia.org/wiki/Leicestershire_County_Cricket_
Club    1622318 /m/05hf4j
beat
Somerset    B   Somerset    Somerset_County_Cricket_Club    http://en.wikipedia.org/wiki/Somerset_County_Cricket_Club   162
2178    /m/05hdty
by
an
innings
and
39
runs
in
two
days
to
take
over
at
the
head
of
the
county
championship
.

Their
stay
on
top
,
though
,
may
be
short-lived
as
title
rivals
Essex   B   Essex   Essex_County_Cricket_Club   http://en.wikipedia.org/wiki/Essex_County_Cricket_Club  1622252 /m/05hdzj
,
Derbyshire  B   Derbyshire  Derbyshire_County_Cricket_Club  http://en.wikipedia.org/wiki/Derbyshire_County_Cricket_Club 182
9984    /m/05_blf
and

[liehe]

My dataset do have these lines, so the dataset should be fine.

[octavian-ganea]

Something is clearly wrong with the p(e|m) index that you use. "perc missing mentions from index : 14.97" is the percentange of mentions m that are not found in the dictionary, while "perc missing entities from mention index : 17.02" is the percentage of gold entities that do not appear in the respective mention entry. These should be together less than 5%. Looking at your log file I see that even common names like "kurdish", "tunisia" or "boston" are missing. Can you please check if they appear in your p(e|m) file (called mek-top-freq-crosswikis-plus-wikipedia-lowercase-top64.txt which should be constructed as a concatenation of the 2 files mek-top-freq-crosswikis-plus-wikipedia-lowercase-top64.txt.part_a*) ?

[octavian-ganea]

This should give a non-empty output:

cat mek-top-freq-crosswikis-plus-wikipedia-lowercase-top64.txt.part_a* | grep -P '^boston\t' | more

namely:

boston  10  10  113041  24437894,85112  167665,4306 65194,3140  43376,1974  69523,691   4339,596    23017869,45
1   882398,417  201767,383  182265,347  730207,324  126401,318  18346514,285    2323878,278 83622,264   550
3022,263    2004519,247 513495,225  1080900,217 211579,207  10128235,199    23876058,190    5637547,179 4608353,159
    876997,158  6721569,150 26372818,150    2593807,142 1692165,139 82254,138   2338329,127 110372,127  288
00877,127   1423832,127 24239512,118    1843613,117 8871435,115 4319938,105 12195659,104    112680,102  206780,10061114,100 621979,98   3387737,92  13436708,91 13832944,90 117682,88   21441922,86 4584803,86  5889873,82230105,68 8305575,67  30055728,60 213128,60   1277059,60  148265,59   1294905,56  23911424,56 206779,53   298047,51   11773015,50 10982304,50 12202877,45 25409421,45

You need to create a new file containing the contents of both files mek-top-freq-crosswikis-plus-wikipedia-lowercase-top64.txt.part_a*, and update its path here: https://github.com/dalab/pboh-entity-linking/blob/master/src/main/scala/index/AllIndexesBox.scala#L19 . Similarly, you need to update the paths of all other index files listed in the same scala file. Let me know if it works.

[liehe]

Thank you so much. It is hard for me to target the problem. I will check the indexes.