search

wasiahmad / PLBART

Official code of our work, Unified Pre-training for Program Understanding and Generation [NAACL 2021].
https://arxiv.org/abs/2103.06333
MIT License
186 stars 35 forks source link

About performance on code summarization #16

Closed LeeSureman closed 3 years ago

LeeSureman commented 3 years ago

Hi, I read your work and I think it's very nice. In recent I run the code summarization experiment follow your readme as follow: bash run.sh 1 [pl_lang] and I got the performance table as follow:

Ruby Javascript Go Python Java PHP
PLBART from paper 14.11 15.56 18.91 19.3 18.45 23.58
PLBART my run 14.36 15.28 18.08 19.92 18.48 23.8

I notice the result of plbart on some pl like "Go" seems lower than paper reported result. Does every programming language have its own hyper-parameters on the task?

wasiahmad commented 3 years ago

No, we actually didn't tune hyper-parameters individually for the languages. So, a slight difference is expected. I think we should run an experiment multiple times (~3) with different seed values and report the average performance.