search

yifding / deep_ed_PyTorch

PyTorch Implementation of deep-ed (EMNLP 2017)
6 stars 1 forks source link

the problem about results #1

Open hu-er opened 2 years ago

hu-er commented 2 years ago

Dear author, thank you very much for recreating deep-ed in Python language, which has greatly helped my research. But now I have a very difficult problem, which is that the deep_ED_Pytorch result is not as same as the result of deep-ed (lua virsion). Here's what I got: wned-aquaint wned-aquaint ; EPOCH = 0: Micro recall = 8418.16% ; Micro F1 = 85.59% wned-msnbc wned-msnbc ; EPOCH = 0: Micro recall = 9039.63% ; Micro F1 = 90.74% wned-ace2004 wned-ace2004 ; EPOCH = 0: Micro recall = 8326.85% ; Micro F1 = 86.12% aida_testB aida_testB ; EPOCH = 0: Micro recall = 8923.08% ; Micro F1 = 89.23% aida_testA aida_testA ; EPOCH = 0: Micro recall = 8837.40% ; Micro F1 = 88.38%

aida_train aida_train ; EPOCH = 0: Micro recall = 9285.91% ; Micro F1 = 92.86% wned-clueweb wned-clueweb ; EPOCH = 0: Micro recall = 7374.04% ; Micro F1 = 73.89% wned-wikipedia wned-wikipedia ; EPOCH = 0: Micro recall = 7614.72% ; Micro F1 = 76.19%

Here's what the paper shows: ==> AQUAINT AQUAINT ; EPOCH = 307: Micro recall = 88.03% ; Micro F1 = 89.51%
==> MSNBC MSNBC ; EPOCH = 307: Micro recall = 93.29% ; Micro F1 = 93.65%
==> ACE04 ACE04 ; EPOCH = 307: Micro recall = 84.05% ; Micro F1 = 86.92%
==> aida-B aida-B ; EPOCH = 307: Micro recall = 92.08% ; Micro F1 = 92.08%
==> aida-A aida-A ; EPOCH = 307: Micro recall = 91.00% ; Micro F1 = 91.01%

I wonder if you have ever encountered this problem? Hope to get your answer and help! I'm so grateful!

hu-er commented 2 years ago

I set the epoch of entity embedding to 200, and the epoch of global model test to 400. My results were obtained by 'ent_vecs__ep_198.pt' and 'global_307.pt' validation.

yifding commented 2 years ago

Hey, thanks for the suggestion. I encounter the same problem which does not 100% repeat the global model performances. Local model and entity embedding seem doing fine. Can we maybe collaborate on implementing the globle model ?

hu-er commented 2 years ago

It seems that this is a problem that still needs to be solved! Let's work together! However, I haven't learned Lua yet, so I hope I can quickly master it and modify the PyTorch version based on the original version. Thank you very much!

yifding commented 1 year ago

the problem should come from the implementation of the global models, I am not quite sure what I implement is exactly the same as the authors report.