search

KLUE-benchmark / KLUE

📖 Korean NLU Benchmark
https://klue-benchmark.com
Creative Commons Attribution Share Alike 4.0 International
554 stars 55 forks source link

YNAT (Where is test data?) #38

Closed conan1024hao closed 2 years ago

conan1024hao commented 2 years ago

Hi, I am a Chinese student studying NLP in Japan and I am also learning Korean. I really love your KLUE website, it's much more beautiful and easy-to-understand than CLUE or JGLUE.

Description

I downloaded data and baseline code from https://klue-benchmark.com/tasks/66/data/download but I found that there is no test data so I could not run the inference script to generate submission file. And there is only inference script without train script, do you have one?

conan1024hao commented 2 years ago

@songys @SungjoonPark @DSKSD @inmoonlight @monologg @warnikchow @myeonghwa-lee Please look at this issue if you have time, thank you.

inmoonlight commented 2 years ago

@conan1024hao We deliberately hide the test set so that the trained model is not easily overfitted on it. You can check the performance on the test set via KLUE leaderboard. Thank you!

conan1024hao commented 2 years ago

@inmoonlight Thank your for reply. Do you mean we need to upload our model file and inference code to use the leaderboard? Since we have no ynat-v1.1_test.json, we could not run inference.py locally.

inmoonlight commented 2 years ago

@conan1024hao Exactly. You can find the example code at https://klue-benchmark.com/tasks/66/data/download

conan1024hao commented 2 years ago

@inmoonlight Sorry to bother you again. I tried upload my model and inference code(which were tested OK locally) as a tar.gz but got submission fail. Since we do not know your computing environment, python version and else, it's really hard to provide codes which can run smoothly. Maybe we will just use your train and dev data for evaluation...