search

salesforce / CodeT5

Home of CodeT5: Open Code LLMs for Code Understanding and Generation
https://arxiv.org/abs/2305.07922
BSD 3-Clause "New" or "Revised" License
2.74k stars 401 forks source link

CodeBLEU #79

Closed Debdeep1998 closed 1 year ago

Debdeep1998 commented 1 year ago

Does CodeBLEU work on generated snippets where the generated code isn't compilable? And if not then how is CodeBLEU used in evaluating CodeT5 model if the generated code by the model might not have an AST.

yuewang-cuhk commented 1 year ago

Hi, from our experience in using CodeBLEU, it can still give reasonable scores when the code is not compilable. For details of how CodeBLEU is computed, please refer to the original paper and we would suggest to ask the original authors for these questions.