search

microth / PathLSTM

Neural SRL model
71 stars 14 forks source link

adding wagon and addresses to release + readme snippet. #11

Closed danyaljj closed 7 years ago

danyaljj commented 7 years ago

with deploying your project, ppl don't need to download your project to compile it. They can just add it as a maven dependency. Here are the deployed files: http://cogcomp.cs.illinois.edu/m2repo/uk/ac/ed/inf/PathLSTM/1.0.0/ I included the maven snippet in your readme file.

microth commented 7 years ago

@danyaljj Can you post this request in the master branch? I am trying to keep the two branches separately because there have been some changes to package names and dependencies over time... :/

danyaljj commented 7 years ago

@microth Marginal question: do you think these changes have changed the performance? Have you evaluated the system after these changes?

microth commented 7 years ago

@danyaljj The idea of this branch is to run the SRL in combination with the CogComp preprocessing chain. So, yes, this will affect performance. I have not yet retrained/tested the model since @christos-c last changes to the dependency parser. I'll hopefully get round to doing that within the next couple of days.

danyaljj commented 7 years ago

I see ... I might try the main branch then. We tried master, and it seems that it's F1 is around 0.83. @bhargav could correct me if I'm wrong.

microth commented 7 years ago

The pretrained model referred to in the README should get you an F1 of 0.879--but only if you use the same preprocessing tools as used for training. If you don't, 0.83 sounds about reasonable. Using the same tools at training and test time makes a big difference in SRL because the model can learn how to correct (systematic) errors by earlier components in the pipeline. That's also why it's still necessary to retrain based on the updated depparse component. But first, my UIUC account needs to be re-activated...

bhargav commented 7 years ago

On the CONLL 2005 test set (Section 23 , WSJ of Penn Treebank), Precision: 83.48 Recall : 79.52 F1 : 81.45

This is using the annotator that Daniel added.

microth commented 7 years ago

My number above referred to CoNLL-2009, which is what the SRL model is trained on. 81.45 on CoNLL-2005 sounds a bit odd to me (the state-of-the-art is 82.xx, see Zhou&Xu'15), because that data set uses a different version of PropBank labels and a different annotation scheme (spans vs. dependencies). @danyaljj Can you confirm that the annotator actually uses my SRL model?

danyaljj commented 7 years ago

I see .... maybe it's the evaluation script that differs? Here is what we use to evaluate.

Bhargav pointed out that this script doesn't penalize predicting too many redundant predicates. I wonder if there is a better way to evaluate it. (@christos-c any thoughts on this?)

danyaljj commented 7 years ago

@danyaljj Can you confirm that the annotator actually uses my SRL model?

It indeed uses your code, although I don't remember which branch. Here is the link to the handler: https://github.com/CogComp/cogcomp-nlp/blob/master/external/src/main/java/edu/illinois/cs/cogcomp/pipeline/handlers/PathLSTMHandler.java

It's here too: http://deagol.cs.illinois.edu:8080/