search

SeoSangwoo / Attention-Based-BiLSTM-relation-extraction

Tensorflow Implementation of "Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification" (ACL 2016)
http://www.aclweb.org/anthology/P16-2034
Apache License 2.0
430 stars 139 forks source link

can`t reach the F1 80.4 #14

Open midnight2104 opened 5 years ago

midnight2104 commented 5 years ago

hi , SeoSangwoo, when i run your code , the result of f1 is 81.56, but the paper of f1 is 84 , how about your socre finally?

heslowen commented 5 years ago

i run this code and the F1 result is only 68%?... i had not change any parameters

ZHOUJessie commented 5 years ago

F1 result is only 68%,but the paper of f1 is 84 , how about your socre finally?

xxxxyan commented 5 years ago

hi , SeoSangwoo, thank you very much for sharing the code firstly , and when i run the code ,i also encountered the problems mentioned above ,the result of f1 is not same as that in paper , i found that PI was not considered in the code,isn't it?

Jerryten commented 5 years ago

Does anyone get the best F1? Is it convenient to reveal the design of super parameters?

xxxxyan commented 5 years ago

Hello,I am sorry to just find this email and can't reply to you timely. I did not reproduce the code of the article either. I would like to ask you a question about the code,

why did use tf.reduce_sum here to reduce dimensions?is it the beat way?I would appreciate it if you could answer my questions.

output = tf.reduce_sum(inputs * tf.expand_dims(alphas, -1), 1)

------------------ 原始邮件 ------------------ 发件人: "Jerry"notifications@github.com; 发送时间: 2019年6月12日(星期三) 下午3:16 收件人: "SeoSangwoo/Attention-Based-BiLSTM-relation-extraction"Attention-Based-BiLSTM-relation-extraction@noreply.github.com; 抄送: "肖燕"592589519@qq.com;"Comment"comment@noreply.github.com; 主题: Re: [SeoSangwoo/Attention-Based-BiLSTM-relation-extraction] can`treach the F1 80.4 (#14)

Does anyone get the best F1? Is it convenient to reveal the design of super parameters?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

wang-h commented 5 years ago

<<< (9+1)-WAY EVALUATION TAKING DIRECTIONALITY INTO ACCOUNT -- OFFICIAL >>>:

Confusion matrix: C-E C-W C-C E-D E-O I-A M-C M-T P-P O <-- classified as +--------------------------------------------------+ -SUM- xDIRx skip ACTUAL C-E | 295 1 0 0 6 1 0 2 2 17 | 324 4 0 328 C-W | 1 245 3 0 0 6 10 5 4 28 | 302 10 0 312 C-C | 0 3 159 8 1 0 2 1 0 17 | 191 1 0 192 E-D | 0 2 5 265 0 0 0 0 0 20 | 292 0 0 292 E-O | 5 2 1 2 225 0 0 1 1 20 | 257 1 0 258 I-A | 3 7 0 2 1 91 0 1 11 39 | 155 1 0 156 M-C | 0 10 1 1 2 0 195 0 2 18 | 229 4 0 233 M-T | 2 5 0 1 1 0 1 217 0 29 | 256 5 0 261 P-P | 4 3 0 0 6 6 1 5 173 31 | 229 2 0 231 O | 22 38 17 31 24 16 33 27 28 218 | 454 0 0 454 +--------------------------------------------------+ -SUM- 332 316 186 310 266 120 242 259 221 437 2689 28 0 2717

Coverage = 2717/2717 = 100.00% Accuracy (calculated for the above confusion matrix) = 2083/2717 = 76.67% Accuracy (considering all skipped examples as Wrong) = 2083/2717 = 76.67% Accuracy (considering all skipped examples as Other) = 2083/2717 = 76.67%

Results for the individual relations: Cause-Effect : P = 295/( 332 + 4) = 87.80% R = 295/ 328 = 89.94% F1 = 88.86% Component-Whole : P = 245/( 316 + 10) = 75.15% R = 245/ 312 = 78.53% F1 = 76.80% Content-Container : P = 159/( 186 + 1) = 85.03% R = 159/ 192 = 82.81% F1 = 83.91% Entity-Destination : P = 265/( 310 + 0) = 85.48% R = 265/ 292 = 90.75% F1 = 88.04% Entity-Origin : P = 225/( 266 + 1) = 84.27% R = 225/ 258 = 87.21% F1 = 85.71% Instrument-Agency : P = 91/( 120 + 1) = 75.21% R = 91/ 156 = 58.33% F1 = 65.70% Member-Collection : P = 195/( 242 + 4) = 79.27% R = 195/ 233 = 83.69% F1 = 81.42% Message-Topic : P = 217/( 259 + 5) = 82.20% R = 217/ 261 = 83.14% F1 = 82.67% Product-Producer : P = 173/( 221 + 2) = 77.58% R = 173/ 231 = 74.89% F1 = 76.21% _Other : P = 218/( 437 + 0) = 49.89% R = 218/ 454 = 48.02% F1 = 48.93%

Micro-averaged result (excluding Other): P = 1865/2280 = 81.80% R = 1865/2263 = 82.41% F1 = 82.10%

MACRO-averaged result (excluding Other): P = 81.33% R = 81.03% F1 = 81.04%

<<< The official score is (9+1)-way evaluation with directionality taken into account: macro-averaged F1 = 81.04% >>> using the default hyper-parameters and small batch size=10, Glove 6d.100.txt

heslowen commented 5 years ago

By using BiLSTM+attention+pytorch ，I got the best marco-F1(exclude relation 'other') 79.2% . I guess that some tricks may lead to a worse result compare to the F1 reported from the paper.

hao notifications@github.com 於 2019年8月27日週二 下午4:28寫道：

I also cannot reach 80%, my best result using Pytorch is only 71.3% (BiLSTM + ATTN), 69.9% (BiLSTM), is the result reported in the paper correct?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SeoSangwoo/Attention-Based-BiLSTM-relation-extraction/issues/14?email_source=notifications&email_token=AFKQGUIKHYL47EVZLAEXWKLQGTQS5A5CNFSM4GKSMIL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5G6FAY#issuecomment-525197955, or mute the thread https://github.com/notifications/unsubscribe-auth/AFKQGULNFNRQZIV5VZZKTS3QGTQS5ANCNFSM4GKSMILQ .

zhijing-jin commented 5 years ago

For everyone's inference,

I ran the code on Sept 9, 2019, did not change anything, and obtained macro-averaged F1 = 81.37%.

<<< The official score is (9+1)-way evaluation with directionality taken into account: macro-averaged F1 = 81.37% >>>

My environment is

Just-silent commented 5 years ago

通过使用BiLSTM + attention + pytorch，我获得了79.2％的最佳marco-F1（排除“其他”关系）。我猜想，与本文报道的F1相比，一些技巧可能会导致更糟糕的结果。hao notifications@github.com于2019年8月27日周二下午4:28时间： … 我也无法达到80％，使用Pytorch的最佳结果仅为71.3％（BiLSTM + ATTN），69.9％（BiLSTM），论文中报告的结果正确吗？—您收到此评论是因为您发表了评论。回复此电子邮件直接，查看它在GitHub < ＃14？email_source =通知＆email_token = AFKQGUIKHYL47EVZLAEXWKLQGTQS5A5CNFSM4GKSMIL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5G6FAY＃issuecomment-525197955>，或静音螺纹< https://github.com/notifications/unsubscribe-auth/AFKQGULNFNRQZIV5VZZKTS3QGTQS5ANCNFSM4GKSMILQ >。

hello,i never used the tensorflow ,can you share your code with me.Rencently,i copy one code from other,which's p can only 62% , i can't find some questions ,so i want study your code by pytorch