search

ntunlp / xCodeEval

xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval
MIT License
71 stars 6 forks source link

Retrieval Tasks Evaluation #6

Closed KOU-201270 closed 1 year ago

KOU-201270 commented 1 year ago

I have noticed that only input data are released in the test set of retrieval tasks. So what steps should I take if I want to evaluate my own model in this task without the corresponding answer? May I use the execEval engine to check if the retrieved candidates can pass the unit test? I would really appreciate it if you could answer this.

sbmaruf commented 1 year ago

No need to use the ExecEval. We will release the gold labels for the retrieval corpus. Our lead author for retrieval task was not available during the final submission. To mitigate the errors, we decided to not release the retrieval gold data. We will release the data shortly. In the meantime, if you need it urgently, please reach out via email.

KOU-201270 commented 1 year ago

Thanks for your kind response. Looking forward to your further update. Good luck.

zlh-source commented 1 month ago

Hello, have the gold labels of the nl_code retrieval dataset been released?