search

hltfbk / Excitement-Open-Platform

Excitement Open Platform for Recognizing Textual Entailments
http://hltfbk.github.io/Excitement-Open-Platform/
86 stars 74 forks source link

3-class entailments? #531

Closed gangeli closed 9 years ago

gangeli commented 9 years ago

Is there a way to use EOP for three class entailment problems (entail / contradict / unknown)? The SICK dataset is released with XML files with 3 classes, but the results seem to only produce "entailment" and "nonentailment".

(Let me know if there's a better place to ask this!)

Thanks!

rzanoli commented 9 years ago

Hello, There are some EDAs that support multi-class problems like BIUTEE and AdArte (I'm not sure about the other EDAs; in case, you would need to ask their developers).

If you want to try AdArte with SICK you can take a look at the Quick Start that will teach you how to use the EOP from the command line (you could start using the platform from the command line to become more confident with the EOP and then you could move using it via its API). For this purpose set AdArte as the EDA to be used instead of EditDistance and SICK as the data set instead of RTE-3. The configuration file of AdArte allows you to try different configurations like the configuration using external resources like WordNet.

Let us know in case of any issues.

Best, Roberto

gangeli commented 9 years ago

Thanks! I found the EDA for AdArte and it runs to completion. However, it does not pick up on the negative and unknown classes. Are the labels for this indeed UNKNOWN and CONTRADICTION as in the sample SICK data?

gangeli commented 9 years ago

I found my error. The scorer only picks up the Entailment and Nonentailment classes, though the output data has the correct 3-class judgments. Would you guys be open to a pull request where I update the scoring script to recognize whatever classes are in the output data?

rzanoli commented 9 years ago

The EOP comes with 2 scorer classes: EDAScorer, and Scorer. The Scorer class should support multi-class issues. If you use the EOP from the command line the EOPRunner class already used the Scorer class for evaluation. In contrast if you use the EOP via API you should use Scorer instead of EDAScorer.