We have no choice: MultipleChoiceTask a few TMMLU Multiple Choice have no choice.

mtkresearch / MR-Models

聯發創新基地（MediaTek Research) 致力於研究基礎模型。我們將研究體現在適合繁體中文使用者的模型上，並在使用權許可的情況下，提供模型給學術界研究或產業界使用。

Other

155 stars 16 forks source link

I have encountered with the evaluation datasets on TMMLU.

Specifically, in the 分科測驗化學 and many more topics dataset, I observed that only question 1 includes choices in the query, whereas the ground truth provides corresponding answers.

I believe addressing this discrepancy is crucial to maintaining the integrity and accuracy of the TMMLU evaluation process. Your prompt attention to this matter would be highly appreciated.

Here's the topic with the same issues:

How and Why does that cause an issue? Because the query prompt doesn't include the choices.

Example QA

Please review this example if you don't believe there is an issue. TMMLU/分科測驗數學甲:3 TMMLU/分科測驗數學甲:4	References	Prompt
B	\ [INST] <\<SYS>> <\<\/SYS>> 15-17題為題組坐標空間中，有一立體Γ的底面位於xy平面上，且底面是由兩拋物線y=x2與 y=8-x2圍成的區域，而Γ的每一個垂直x軸的截面都是等腰直角三角形，且三角形的斜邊在xy平面上。設每一個垂直x軸且通過2(t ,t2,0)的截面所成等腰直角三角形面積為 A(t) 。試回答下列問題。【題組】15. 試求此立體Γ的底面面積。 (4 分 ) [/INST]
D	\ ~~[INST] <\> <\<\/SYS>> 【題組】16.試以t表示A(t)。(2分) [/INST]~~

Please review this example if you don't believe there is an issue. TMMLU/分科測驗數學甲:3 TMMLU/分科測驗數學甲:4

References

Prompt

\ [INST] <\<SYS>> <\<\/SYS>> 15-17題為題組坐標空間中，有一立體Γ的底面位於xy平面上，且底面是由兩拋物線y=x2與 y=8-x2圍成的區域，而Γ的每一個垂直x軸的截面都是等腰直角三角形，且三角形的斜邊在xy平面上。設每一個垂直x軸且通過2(t ,t2,0)的截面所成等腰直角三角形面積為 A(t) 。試回答下列問題。【題組】15. 試求此立體Γ的底面面積。 (4 分 ) [/INST]

\ ~~[INST] <\> <\<\/SYS>> 【題組】16.試以t表示A(t)。(2分) [/INST]~~

mtkresearch / MR-Models

We have no choice: MultipleChoiceTask a few TMMLU Multiple Choice have no choice. #15