microsoft / CodeXGLUE

CodeXGLUE
MIT License
1.51k stars 363 forks source link

All ground truths are missing in line level code completion `test.json` files #110

Closed felixzli closed 2 years ago

felixzli commented 2 years ago

In dataset/py150/line_completion/test.json and dataset/javaCorpus/line_completion/test.json, every line's json object has empty "gt": ""

For example, here is the first line in dataset/py150/line_completion/test.json

{
  "id": 51,
  "input": "<s> import threading <EOL> import IECore <EOL> import Gaffer <EOL> import GafferUI <EOL> import GafferImage <EOL> __all__ = [ ] <EOL> Gaffer . Metadata . registerNode ( <EOL> GafferImage . Display , <EOL> \"<STR_LIT:description>\" , <EOL> \"\"\"<STR_LIT>\"\"\" , <EOL> plugs = { <EOL> \"<STR_LIT:port>\" : [ <EOL> \"<STR_LIT:description>\" , <EOL> \"\"\"<STR_LIT>\"\"\" , <EOL> ] , <EOL> } <EOL> ) <EOL> __plugsPendingUpdate = [ ] <EOL> __plugsPendingUpdateLock = threading . Lock ( ) <EOL> def __scheduleUpdate ( plug , force = False ) : <EOL> if not force : <EOL> global __plugsPendingUpdate <EOL> global __plugsPendingUpdateLock <EOL> with __plugsPendingUpdateLock : <EOL> for p in __plugsPendingUpdate : <EOL> if plug . isSame ( p ) : <EOL> return <EOL> __plugsPendingUpdate . append ( plug ) <EOL> GafferUI . EventLoop . executeOnUIThread ( lambda : __update ( plug ) ) <EOL> def __update ( plug ) : <EOL> node = plug . node ( ) <EOL> if node : <EOL> updateCountPlug = node [ \"<STR_LIT>\" ] <EOL> updateCountPlug . setValue ( updateCountPlug . getValue ( )",
  "gt": ""
}

According to the README.md at https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeCompletion-line ,"gt" should not be empty. image

felixzli commented 2 years ago

https://github.com/microsoft/CodeXGLUE/issues/102 ground truth of test set in line completion task is not provided submit predictions to codexglue@microsoft.com to get results