search

microsoft / CodeXGLUE

CodeXGLUE
MIT License
1.57k stars 366 forks source link

Text-Code/NL-code-search-WebQuery/ | Lack of webquery_answers.txt in #79

Closed KevinHuuu closed 3 years ago

KevinHuuu commented 3 years ago

Hi there, awesome project! Thanks for open source it for the community!

I am working on a new dataset, and trying to use it to augment current provided dataset during finetuning on several CodeXGLUE tasks. According to the readme file in

"After generate predictions for WebQueryTest, you can use our provided script to evaluate:"

python evaluator/evaluator.py \
        --answers_webquery ./evaluator/webquery_answers.txt \
        --predictions_webquery evaluator/webquery_predictions.txt

It seems the webquery_answers.txt is not provided. Do we actually can not evaluate the task by ourselves due to the lack of answers.txt data?

Thank you!

celbree commented 3 years ago

Hi, we are not going to reveal the ground truth of test set. You can submit your predictions to codexglue@microsoft.com.