search

kentonl / e2e-coref

End-to-end Neural Coreference Resolution
Apache License 2.0
518 stars 174 forks source link

Conll Scores for .jsonlines #22

Closed MarkusKrug closed 6 years ago

MarkusKrug commented 6 years ago

Hi, first of all thank you for the release of the code. I managed to train on my own data and got a bunch of models. My next step would be to get the conll scores, however i only got my test document converted to .jsonlines. Is there any easy way to get the conll scores for those?

herbertchen1 commented 6 years ago

it's in the evaluator.py and you can test with conll.pl???

MarkusKrug commented 6 years ago

Thanks fo the answer, but here is my problem with some more details:

  1. I converted my test documents into a single test.jsonlines.
  2. When i call evaluator.py, the a model from the trained ones is used and evaluated. I also receive reasonable mention evaluation output.
  3. Then when the eval script tends to try and calculate the conll scores for my .jsonline data i always only get an error. 
    
ValueError: invalid literal for int() with base 10: '"1","```

So from this point (and since i did no dig deep enough into the code) i dont know if the conll eval does indeed only work for connl data and i have to use a workaround, like predicting using the demo and performing an evaluation from there.

MarkusKrug commented 6 years ago

Okay im able to evaluate it now by batch predicting and performing the eval on the result of that which works fine!