search

BBN-E / tac-kbp-eal

TAC KBP Event Argument Extraction and Linking Shared Task
MIT License
25 stars 19 forks source link

Unclear Error Message #8

Open rktilia opened 7 years ago

rktilia commented 7 years ago

When I run evaluate_2016.sh, the script crashes and I am getting the following error message:

java.lang.RuntimeException: The set of linked responses should exactly equal the set of non-generic argument responses. However,
The following are in the linking only:
 NYT_ENG_20130630.0084-Contact.Contact-Entity([1549:1551]=He[1549-1551]; Generic; [1657-1671]; [])
NYT_ENG_20130630.0084-Contact.Contact-Entity([1565:1571]=people[1565-1571]; Generic; [1657-1671]; [])
NYT_ENG_20130630.0084-Contact.Contact-Entity([1651:1656]=their[1651-1656]; Generic; [1657-1671]; [])
NYT_ENG_20130630.0084-Contact.Broadcast-Audience([2699:2701]=It[2699-2701]; Generic; [2714-2721]; [])
NYT_ENG_20130630.0084-Contact.Broadcast-Audience([2714:2721]=release[2714-2721]; Generic; [2714-2721]; [])
    at com.bbn.kbp.events2014.bin.ValidateSystemOutput.checkLinkingValidity(ValidateSystemOutput.java:339)
    at com.bbn.kbp.events2014.bin.ValidateSystemOutput.validate(ValidateSystemOutput.java:227)
    at com.bbn.kbp.events2014.bin.ValidateSystemOutput.validateOnly(ValidateSystemOutput.java:121)
    at com.bbn.kbp.events2014.bin.ValidateSystemOutput2016.trueMain(ValidateSystemOutput2016.java:97)
    at com.bbn.kbp.events2014.bin.ValidateSystemOutput2016.main(ValidateSystemOutput2016.java:37)

I cannot seem to figure out what exactly it means and what is going wrong. I would be good to get some advice on this.

Thanks a lot!

jayded commented 7 years ago

The error means what it says - you can't link generic responses together, so having them in your linking store is an error. See the 2016 guidelines at https://tac.nist.gov//2016/KBP/Event/Argument/guidelines.html

rktilia commented 7 years ago

Ok thanks! I managed to solve this issue. In my opinion it is really missleading that it says The following are in the linking only. This seems to imply that they are expected to also appear somewhere else which is not the case. It might make sense to change that error message to something like "The following should not occur in the linking" to make it a bit more understandable.