search

attardi / deepnl

Deep Learning for Natural Language Processing
GNU General Public License v3.0
457 stars 116 forks source link

SENNA distribution missing file #16

Closed dungtn closed 8 years ago

dungtn commented 8 years ago

Hi,

I cannot reproduce the result listed, instead I got this processed 46435 tokens with 5648 phrases; found: 5642 phrases; correct: 4576. accuracy: 96.04%; precision: 81.11%; recall: 81.02%; FB1: 81.06 LOC: precision: 79.34%; recall: 88.19%; FB1: 83.53 1854 MISC: precision: 68.24%; recall: 67.95%; FB1: 68.09 699 ORG: precision: 77.88%; recall: 69.96%; FB1: 73.71 1492 PER: precision: 91.80%; recall: 90.66%; FB1: 91.23 1597

I have strictly followed your instruction, except for the senna/hash/words.txt wasn't found, so I use senna/hash/words.lst instead, is that ok?

attardi commented 8 years ago

words.lst is OK. The results reported were obtained using an older version available here: https://github.com/attardi/nlpnet that did not use AdaGrad and used different learning rate parameters for different features. It also did not set the random seed, hence the results varied at each run. DeepNL instead sets the seed in dl-ner.py for replicable results. You might try disabling AdaGrad by using parameter --eps 0. However an FB1 of 81% is definitely too low.

dungtn commented 8 years ago

I have re-trained the model with the same settings and got 84%, I will run a 10-fold cross validation and and post the result here (take time). I will also try your suggestions and see if it helps :+1:

cerisara commented 8 years ago

I have tested version 78f6e without random seed init on Conll03 but using my own random init in java within the same interval [+/-2.38... as in the python code], and ran it 10 times, evaluate with the conlleval script on testb and obtain an average F1=88.12%, standard deviation=0.20% With the python random seeded init, I got F1=88.42%, so it's ok I think