About data preprocessing for APPS

[Switchsyj]

Hi, thanks for your amazing work. I wonder how to organize train/test split with APPS dataset as xlcost format. Could you share preprocessing scripts with us, thanks!

[Switchsyj]

And any instructions about how to load_base_model in model.py line 19? (i.e. Did I miss any checkpoint files which should be organized under baselines/saved_models/java-cpp/?)

[parshinsh]

@Switchsyj For the APPS dataset, we follow the same preprocessing step as used in the original paper. Please check here and use apps_create_split.py to create the train and test .json files.

[parshinsh]

@Switchsyj What we mean by the model saved in the path baselines/saved_models/java-cpp/ is the base CodeT5 model (before RL) that has been fine-tuned on the java-cpp translation task using the XLCoST dataset. Since there are multiple fine-tuned models for translation on different language pairs, we have not released the checkpoints. You can obtain this fine-tuned model checkpoints for each language pair by using run_translation.sh script on the original XLCoST github repo here.

[Switchsyj]

Thanks for your answering, another trivial question is that if I want to re-implement your text2code results in Table3, then I need to prepare fine-tuned codeT5 model by running bash run_NL_PL.sh 2 comment python snippet codet5 train here. Then evaluate pass@k the same as human-eval. Am I correct?

[parshinsh]

@Switchsyj The first link that you have provided is not working but I think you got the logic correctly. You need to first fine-tune the codeT5 model on the APPS dataset, following the same approach as you would for translation task scripts.