search

ICSE-DOME / DOME

Developer-Intent Driven Code Comment Generation
15 stars 1 forks source link

Accuracy on the comment classification does not reach the reported value? #1

Open anoynomous101010 opened 1 year ago

anoynomous101010 commented 1 year ago

Thanks for your great work. I have tried to train the comment classifier based on your instrutions., i.e., finetuned the dataset using CodeBert. However, the test results on the test set are: image which does not reach the accuracy repoted in the paper. Could you please give some advice or release the pkl file of the comment classifier? Looking forward to your reply!

ICSE-DOME commented 1 year ago

Hi, for the comment classifier, we manually annotate 20k code-comment data into six intent categories. Since some comments are very vague, and it is difficult to determine their intent, we remove these comments (i.e. noisy data) and split the rest of the data into 10 folds (9 folds for training and 1 fold for testing).

I have uploaded the 10-fold dataset and the well-trained checkpoint file on Google Drive, you can download them from this link https://drive.google.com/file/d/15WUpuxyzy21nTuECvL4BFxYbCOihue6b/view. The results on the test set should be: image

anoynomous101010 commented 1 year ago

Hi, thanks for the released pkl file and the dataset. I still have a question about the dataset. Is the 10-fold dataset finally pre-processed, i.e., clean version? And what is the final amount of training and test data in your dataset? I found that in the "label" attribute of the train_0.json, there are some values "convert", not belonging to the six categories. Could you please check and help me clarify? Thanks in advance!

hungkien05 commented 10 months ago

@anoynomous101010 Hi, have you found any solutions for this problem ? Besides "convert", I also notice there are labels named "what-why" and "what-how" .