lgw863 / LogiQA-dataset

105 stars 10 forks source link

English translation has lots of issues #9

Open santiontanon opened 2 years ago

santiontanon commented 2 years ago

Hi, I was evaluating models on the English version, and I noticed they did very poorly. So, I decided to look at the questions myself by hand, and I cannot answer them either, since, at least the 4-5 I looked at all have translation errors that make the questions un-answearable. Specially, in the answers, which contain small snippets of text, some times the translations do not make any sense at all. I wonder if there are any plans for a revised translation.

diziet commented 9 months ago

I agree. For example, here is are two randomly sampled questions:

b
In a traditional Chinese medicine preparation, there must be at least one kind of ginseng or codonopsis, and the following conditions must also be met? 1) If there is codonopsis, there must be atractylodes.2) Atractylodes macrocephala and ginseng can only have at most one.You must have Shouwu.4) If you have Shouwu, you must have Atractylodes.
According to the above statement, which of the following can be drawn about this Chinese medicine preparation?
A No dangshen
B No Shouwu
C 有 白 术
D 不 白 术

Here's another one:

d
After multiple rounds of elimination, four players A.B, C and D compete for the final ranking.The ranking does not have a parallel ranking.Analysts predict? I, the first place is either A or B; II.If C Not the first, Ding is not the first; III, A is not the first.
If only one sentence of the analyst ’s prediction is correct, who is the first?
A.C
B.B
C.Can't push
D.Ding

Perhaps v2 of the dataset is better? https://github.com/csitfun/LogiQA2.0/blob/main/logiqa/DATA/LOGIQA/test.txt