This PR replaces the English words entailment, contradiction, and neutral in the prompt for the JNLI task with the Japanese words 含意, 矛盾, and 中立. The purpose behind this is to guarantee that we evaluate the Japanese capability of the language models (and not necessarily English capabilities).
Note that the task version has been bumped from 1.1 to 1.2, since this modifies the task definition.
Details
The following table shows the balanced accuracy for JNLI across several models.
As shown, prior to this PR, many models have a score of 33.33, which is essentially the same as random chance (given that there are 3 classes to choose from). After this PR, none of the models have a score of 33.33.
Overview
This PR replaces the English words
entailment,contradiction, andneutralin the prompt for the JNLI task with the Japanese words含意,矛盾, and中立. The purpose behind this is to guarantee that we evaluate the Japanese capability of the language models (and not necessarily English capabilities).Note that the task version has been bumped from 1.1 to 1.2, since this modifies the task definition.
Details
The following table shows the balanced accuracy for JNLI across several models. As shown, prior to this PR, many models have a score of 33.33, which is essentially the same as random chance (given that there are 3 classes to choose from). After this PR, none of the models have a score of 33.33.