search

dmis-lab / BioSyn

ACL'2020: Biomedical Entity Representations with Synonym Marginalization
https://arxiv.org/abs/2005.00239
MIT License
160 stars 26 forks source link

Reproducibility of results on the NCBI Disease corpus #5

Closed ArnaudFerre closed 3 years ago

ArnaudFerre commented 3 years ago

Hi,

Thanks to your Git repo, I was able to make 10 runs on the NCBI corpus with BioSyn. I only did a single parsing of the data, then 10 independent preprocessing + training + prediction. My results (with your evaluation script) are Acc1=89.89 with a standard deviation of 0.64 (the variability of the results coming almost exclusively from the preprocessing, from Ab3P I guess).

I understand from your article that your Acc1=91.1 result is obtained on a single run, right? If so, it seems consistent.

If not, would you have an idea where the small difference could come from? (in this case, I could provide my bash commands, but they are basically a copy of the ones in your readme, without any modification of the options)

Kind regards

mjeensung commented 3 years ago

Hi, ArnaudFerre

Yes, I obtained the result on a single run.

Best regards

arka2010 commented 3 years ago

How did you get the results in one run. Please feel free to demonstrate.

Regards Arka

On Thu, 13 May 2021, 05:04 Mujeen Sung, @.***> wrote:

Hi, ArnaudFerre

Yes, I obtained the result on a single run.

Best regards

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dmis-lab/BioSyn/issues/5#issuecomment-840169812, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGKWOH6X56CZAO7ZPBXTOCLTNMGCBANCNFSM44WPCXHA .