Slow inference speeds of CodeBERT and UniXCoder models

[MayankAgarwal]

Hi, thanks for providing this code base, it is a valuable resource for all of us.

While trying to evaluate the CodeBERT and UniXCoder models on the validation set, I noticed that generation takes a long time (as compared to other models I have been working with). In my experiments, on an A100 GPU it takes about 90 seconds for a batch of 8 samples to generate with a beam size of 10 and 512 target tokens. For a test set of 1000 examples, it would take the model over 2 hours just to generate the samples.

Did you notice similar speeds in your experiments as well, and do you have any pointers on how this could be sped up? I noticed that in the paper you use smaller target tokens, so maybe that could be an issue too.

[guoday]

Yes, our generation is quite slow because we are performing beam search on the data one instance at a time. It needs to rewrite the beam search code to speedup.

[MayankAgarwal]

Thank you for confirming.