Ontonotes train/test/split

[kamalkraj]

I unable get the results specified in your paper using the train/test/dev split using conll2012 v4

Which version did you use ?

[strubell]

Can you be more specific about which results you tried to replicate?

I used the data here: http://conll.cemantix.org/2012/data.html which is train-v4, dev-v4 and test-v9.

[kamalkraj]

Data we both used are same, I tried to replicate the results from your paper Fast and Accurate Entity Recognition with Iterated Dilated Convolutions On the test I got F1 score 64 , but in the paper F1 score is around 86

[strubell]

Ok. Which config are you using?

On Wed, Mar 14, 2018 at 3:17 PM Kamal Raj notifications@github.com wrote:

Data we both used are same, I tried to replicate the results from your paper Fast and Accurate Entity Recognition with Iterated Dilated Convolutions On the test I got F1 score 64 , but in the paper F1 score is around 86

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/iesl/dilated-cnn-ner/issues/10#issuecomment-373143171, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHZt7ECxCLPBt53YN4NoPvesAN_FBUSks5teWzOgaJpZM4Sq0k3 .

[strubell]

I also don't use Collobert embeddings in that paper (which is what I am guessing you're using due to the other issue you opened). Try with these embeddings: https://drive.google.com/open?id=1AQX_RGIc7Sopyi3cGEYBfbKV848VXzJn

On Wed, Mar 14, 2018 at 3:20 PM Emma Strubell emma.strubell@gmail.com wrote:

Ok. Which config are you using?

On Wed, Mar 14, 2018 at 3:17 PM Kamal Raj notifications@github.com wrote:

Data we both used are same, I tried to replicate the results from your paper Fast and Accurate Entity Recognition with Iterated Dilated Convolutions On the test I got F1 score 64 , but in the paper F1 score is around 86

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/iesl/dilated-cnn-ner/issues/10#issuecomment-373143171, or mute the thread https://github.com/notifications/unsubscribe-auth/ADHZt7ECxCLPBt53YN4NoPvesAN_FBUSks5teWzOgaJpZM4Sq0k3 .

[ghaddarAbs]

I tested the dilated-cnn config ...... it seems correct to me.... not that far away :)

Best dev F1: 84.55

Segment evaluation (test):
            F1  Prec    Recall
Micro (Seg) 85.37   85.60   85.14
Macro (Seg) 72.14   73.70   70.65
-------
   ORDINAL  80.31   80.10   80.51
       LOC  69.92   67.89   72.07
   PRODUCT  52.70   54.17   51.32
      NORP  92.38   91.78   92.98
WORK_OF_ART 48.52   47.67   49.40
  LANGUAGE  44.44   57.14   36.36
     MONEY  83.68   83.28   84.08
   PERCENT  88.32   87.82   88.83
    PERSON  90.85   91.57   90.14
       ORG  83.37   83.79   82.95
  CARDINAL  82.71   83.85   81.60
       GPE  93.79   94.60   92.99
      TIME  58.39   60.30   56.60
      DATE  83.21   81.58   84.89
       FAC  62.17   62.88   61.48
       LAW  61.76   75.00   52.50
     EVENT  45.87   54.35   39.68
  QUANTITY  70.97   68.75   73.33
Processed 152728 tokens with 11257 phrases; found: 11196 phrases; correct: 9584.
Testing time: 44 seconds

[kamalkraj]

@strubell One last question F1 score presented in the Fast and Accurate Entity Recognition with Iterated Dilated Convolutions paper is F1_macro or F1_micro ?

[ghaddarAbs]

Micro

[strubell]

Thanks!

[blester125]

Hello, I have a question about the datasplit. From your earlier comment.

I used the data here: http://conll.cemantix.org/2012/data.html which is train-v4, dev-v4 and test-v9.

In test-v9 as far as I can tell there are 11,057 entities while in the test-v4 section there are 11,257 like you mention in Table 8. Which test set did you use? From the look of the preprocess.sh script it lookes like it was the v4 because the names of the files are gold_conll rather than gold_parse_conll like in the v9 test set.

I used this to find entites find annotations/ -name '*.v4_gold_conll' | grep -v 'pt/nt' | xargs cat | sed 's/\s\s*/ /g' | cut -d' ' -f11 | sed -n '/^(/p' | wc -l

Thanks for your clarification.