search

microsoft / CodeXGLUE

CodeXGLUE
MIT License
1.55k stars 366 forks source link

There is no answer data in ClozeTesting part? #56

Closed ChrisCN97 closed 3 years ago

ChrisCN97 commented 3 years ago

Hi, I want to make some experiments based on the Cloze Test task. But it seems that there is no answer data in both ClozeTesting-all and ClozeTesting-maxmin in this project. According to evaluator.py, the calculate_scores function needs the answers.txt and the model predicting results predictions.txt to calculate the final scores. But all the evaluator/answers/\/answers.txt are like: maxmin-1<CODESPLIT>token1 maxmin-2<CODESPLIT>token2 maxmin-3<CODESPLIT>token3 maxmin-4<CODESPLIT>token4 maxmin-5<CODESPLIT>token5 I do not understand what it means. I think the answers.txt should contains the real answers corresponding to the cloze-maxmin/\/clozeTest.json. Is there somewhere else I can access the related data?

Thanks.

Jun-jie-Huang commented 3 years ago

Hi, thanks for your interest in our dataset. Please refer to issue #38 and #20.

ChrisCN97 commented 3 years ago

thanks