Clarification on Datasets and Hyperparameters Used in Experiments

[pppyb]

I am currently trying to reproduce the experimental results described in your repository and have a few questions regarding the specifics of your setup. I noticed that the source code references the dataset datasets/random-api-completion.test.jsonl. Could you please confirm if this dataset was used for all the models evaluated in the experiments as in the paper's results, or were there different datasets used for each model?

Additionally, it would be very helpful if you could provide details on the hyperparameter settings for the models, as well as any specific configurations crucial for replication. I am also interested in the inference code used to generate the results. Could you please point me to the relevant scripts or provide some examples of how to set up the inference environment correctly?

Thank you for your assistance and for sharing your work. I look forward to your response!

[zfj1998]

please refer to this pull request (https://github.com/microsoft/CodeT/pull/20) for the dataset path matter. We rename the datasets/random-api-completion.test.jsonl to another name when uploading the data files.

For the inference hyperparameters, we use greedy search and a maximum of 100 new tokens to generate. I have updated the inference code for codegen model in this pull request (https://github.com/microsoft/CodeT/pull/20/commits/cecce36b082e0e78c0c4097a7cb4777e5e7d5f28).

Thanks for your interest in our work!