search

neulab / code-bert-score

CodeBERTScore: an automatic metric for code generation, based on BERTScore
MIT License
172 stars 16 forks source link

Getting IndexError when any of 'cands', 'refs' and 'sources' contains empty string #4

Open terryyz opened 1 year ago

terryyz commented 1 year ago

Current code_bert_score is able to handle the case where any of 'cands', 'refs' contains empty strings and no 'sources' is passed to the score method. See the example below:

from code_bert_score import score
score([''],['a'], lang="python")

However, when 'sources' is provided, the method will raise IndexError.

from code_bert_score import score
score([''],['a'], sources=["a"], lang="python")

It would be great if this kind of cases can be handled.

urialon commented 1 year ago

Thanks @terryyz ! I just pushed a fix.

In case the cand or ref is an empty string, I am now setting both precision and recall to zero. It is now working when sources are provided or not provided.

Please let us know if you have any questions or find any other issues! Best, Uri

littlepopy commented 11 months ago

Can I ask you the data provided under code-bert-score/evaluation come from where? The generation codes ### _(humanevalhyps.txt) of 13025 lines, with the corresponding NL_(humanevalsrcs.txt) and reference code_(humanevalrefs.txt) . And if we want to use the data, is it be promised?

terryyz commented 11 months ago

Hi @littlepopy, the details were discussed in Appendix B.2 in the paper, where the data was collected from HumanEval-X. Besides, you may also find our paper useful. This folder may provide all the data you need 😃

urialon commented 11 months ago

Hi @littlepopy , Thank you for your interest in our work.

Please see the paragraph "functional correctness experiments" in Section 3.2 in the paper https://arxiv.org/pdf/2302.05527.pdf

And also, Appendix B.2.

Let us know if you have any questions! Uri

littlepopy commented 11 months ago

Thanks for your reply. It is very useful. Besides, thanks for all the work you have done. Wish you a good future.

------------------ 原始邮件 ------------------ 发件人: "Uri @.>; 发送时间: 2024年1月4日(星期四) 下午4:49 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [neulab/code-bert-score] Getting IndexError when any of 'cands', 'refs' and 'sources' contains empty string (Issue #4)

Hi @littlepopy , Thank you for your interest in our work.

Please see the paragraph "functional correctness experiments" in Section 3.2 in the paper https://arxiv.org/pdf/2302.05527.pdf

And also, Appendix B.2.

Let us know if you have any questions! Uri

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

littlepopy commented 11 months ago

Thanks for your reply. It is very useful. Besides, thanks for all the work you have done. Wish you a good future.

------------------ 原始邮件 ------------------ 发件人: "Terry @.>; 发送时间: 2024年1月4日(星期四) 下午4:48 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [neulab/code-bert-score] Getting IndexError when any of 'cands', 'refs' and 'sources' contains empty string (Issue #4)

Hi @littlepopy, the details were discussed in Appendix B.2 in the paper, where the data was collected from [HumanEval-X](https://huggingface.co/datasets/THUDM/humaneval-x]. Besides, you may also find our paper useful. This folder may provide all the data you need 😃

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>