The results do not match those reported in the paper.

[ikechiyo]

Hello, I used the runcommits\<encoding>.sh script from the replication package to conduct experiments on AspectJ, but the accuracy was lower than the results reported in Table 2 of the paper. Are the hyperparameters specified in the script the same as those used in the evaluation experiments?

[X-Luminous]

Hello, I used the runcommits.sh script from the replication package to conduct experiments on AspectJ, but the accuracy was lower than the results reported in Table 2 of the paper. Are the hyperparameters specified in the script the same as those used in the evaluation experiments?

I have the same problem. Have you solved your problem? If so, can you tell me how you set the parameters?

[duyali2000]

The hyperparameters for the experiments are selected based on the model's performance across different datasets and may vary accordingly. I would recommend testing different hyperparameter configurations to identify the optimal setup for your use case.

Additionally, maybe a small trick we discovered during our experiments is that training at the commit-level tends to converge faster and is less prone to oscillations. You could try initializing the models at the file-level and hunk-level with the weights obtained from the commit-level model training. This trick is effective in accelerating convergence and improving performance in our experiments.

If you have other questions, feel free to email me at duyl@lamda.nju.edu.cn.