search

YoungXiyuan / DCA

This repository contains code used in the EMNLP 2019 paper "Learning Dynamic Context Augmentation for Global Entity Linking".
https://arxiv.org/abs/1909.02117
46 stars 15 forks source link

Please provide detailed parameters to reproduce the results in Table 2 and 3 in the paper #1

Closed hitercs closed 5 years ago

hitercs commented 5 years ago

Hi,

Thanks for your work. I run your code with the default arguments by following your command in the readme.

Supervised Learning: python main.py --mode train --order offset --model_path model --method SL

Reinforcement Learning: python main.py --mode train --order offset --model_path model --method RL

After running for 500 epoches, I got results which is much lower than the paper's report results. Specially, in SL setting: best_aida_A_rlts [['aida-A', 0.9149358104581986], ['aida-B', 0.9279821627647714], ['msnbc', 0.9303749043611323], ['aquaint', 0.8433566433566434], ['ace2004', 0.8772635814889336], ['clueweb', 0.7220625224577794], ['wikipedia', 0.7024628355890836]]

In RL setting: best_aida_A_rlts [['aida-A', 0.911178373864941], ['aida-B', 0.9208472686733556], ['msnbc', 0.9319051262433052], ['aquaint', 0.8657342657342657], ['ace2004', 0.8812877263581488], ['clueweb', 0.7107438016528925], ['wikipedia', 0.7360402337105243]]

The default order is offset, but in Fig.3 DCA-SL offset should be 94.35, while DCA-RL should be 93.70 on AIDA-B.

So could you please provide the full command including detailed arguments setting to reproduce the results?

Thanks a lot. Looking forward to your reply.

YoungXiyuan commented 5 years ago

Thank you for your attention to our work.

First, it is indeed our mistake to provide incomplete parameters, for that so many experiments in ablation study disturbs the original best parameter setting.

Second, it seems that you just adopt our Berkeley-CNN baseline, because the provided results in E-mail are similar to our reported results of Berkeley-CNN + DCA-(SL/RL) in the paper.

I would tell you how to switch to ETHZ-Attn baseline later.

Best

------------------ Original ------------------ From: "Chen Shuang";notifications@github.com; Send time: Saturday, Oct 12, 2019 3:02 PM To: "YoungXiyuan/DCA"DCA@noreply.github.com; Cc: "Subscribed"subscribed@noreply.github.com; Subject: [YoungXiyuan/DCA] Please provide detailed parameters to reproduce theresults in Table 2 and 3 in the paper (#1)

Hi,

Thanks for your work. I run your code with the default arguments by following your command in the readme.

Supervised Learning: python main.py --mode train --order offset --model_path model --method SL

Reinforcement Learning: python main.py --mode train --order offset --model_path model --method RL

After running for 500 epoches, I got results which is much lower than the paper's report results. Specially, in SL setting: best_aida_A_rlts [['aida-A', 0.9149358104581986], ['aida-B', 0.9279821627647714], ['msnbc', 0.9303749043611323], ['aquaint', 0.8433566433566434], ['ace2004', 0.8772635814889336], ['clueweb', 0.7220625224577794], ['wikipedia', 0.7024628355890836]]

In RL setting: best_aida_A_rlts [['aida-A', 0.911178373864941], ['aida-B', 0.9208472686733556], ['msnbc', 0.9319051262433052], ['aquaint', 0.8657342657342657], ['ace2004', 0.8812877263581488], ['clueweb', 0.7107438016528925], ['wikipedia', 0.7360402337105243]]

The default order is offset, but in Fig.3 DCA-SL offset should be 94.35, while DCA-RL should be 93.70 on AIDA-B.

So could you please provide the full command including detailed arguments setting to reproduce the results?

Thanks a lot. Looking forward to your reply.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

hitercs commented 5 years ago

Thanks for your reply! Waiting for your further update. ps: please also add detailed parameters settings for your best results.

Thanks.

Best

hitercs commented 5 years ago

Hi,

Any updates on this Issue?

Thanks a lot. Best

StephanieWyt commented 5 years ago

Hi, I have the same problem with @hitercs about reproducing the reported best results. Looking forward to your early reply.

Many thanks. Best

YoungXiyuan commented 5 years ago

Hi, everyone. Sorry for later reply!

Now we are sure that your provided results is produced by our Berkeley-CNN baseline. (See Line # 125 and #295 in the mulrel_ranker.py)

We are trying to modify our original code, which would take one or two days.

Please excuse this clerical error.

hitercs commented 5 years ago

@YoungXiyuan Thanks for your quick reply! Got it. That's fine.

Best

StephanieWyt commented 5 years ago

@YoungXiyuan Thanks for your quick response much appreciated.

Best

YoungXiyuan commented 5 years ago

Hi, everyone. We now update the "mulrel_ranker.py" file, and find that the default parameter setting in the "main.py" should be sufficient to reproduce results reported in our paper.

So please download the newly updated "mulrel_ranker.py" file and try again.

Best

hitercs commented 5 years ago

@YoungXiyuan , Great! Thanks.

Best

hitercs commented 5 years ago

I can reproduce the results now. Good work. close this issue.