FudanSELab / ClassEval

Benchmark ClassEval for class-level code generation.
MIT License
123 stars 8 forks source link

Several minor issues when running tests on ClassEval. #2

Closed ztwater closed 9 months ago

ztwater commented 9 months ago

Thanks for your contribution with a class-level context-dependent method generation benchmark. Here are some minor problems I found when I inspected the test results. I downloaded this benchmark in August and maybe some of them have been fixed up to now.

  1. 测试方法中所依赖的python包需要提前安装,建议可以给一个list,否则会导致运行测试时大量的NameError。

  2. ClassEval_5_display 返回值应该是string而非None

  3. ClassEval_44_format_line_html_text 测试代码中多了from benchmark_code.HtmlUtil import HtmlUtil

  4. ClassEval_49_search_jobs :param criteria: The requirements of the position,list. 实际上输入是字符串,e.g., 'skill1'

  5. ClassEval_50 类名是JSONProcessor,实际文件命名为JSONHandler

  6. ClassEval_52_lemmatize_sentence "solution_code"删去nltk download语句,下载nltk对应包,放到环境中,防止测试超时的问题

  7. ClassEval_60_create_table 方法描述中的字段名写错,'author name'应该是'theater_name'

  8. ClassEval_69_extract_text_from_pdfs 方法的测试代码少了一个测试类

  9. ClassEval_94_display_items :return: If the vending machine is empty, returns False, otherwise, returns a list of the products in the vending machine, list. 实际测试用例写的是返回 string

DXY-lemon commented 9 months ago

非常感谢您的反馈!我们会仔细检查这些问题并且重新发布升级后的benchmark以及所需的list

DXY-lemon commented 9 months ago

我们已经修复您提到的问题,重新发布了1.1.0版本的classeval数据,并且更新了requirements

ztwater commented 9 months ago

感谢你们的迅速回复和处理!ClassEval是一个对领域很有价值的benchmark,期待它的影响力越来越大。