I'd like to test FIM capability of my model,does this test set have a fim subset?(i.e.function list depends on the following context)

FudanSELab / ClassEval

Benchmark ClassEval for class-level code generation.

MIT License

108 stars 5 forks source link

I'd like to test FIM capability of my model,does this test set have a fim subset?(i.e.function list depends on the following context) #1

Closed zndqs closed 4 months ago

zndqs commented 10 months ago

我想要测试模型的fim能力，这个测试集有fim的子集吗（即依赖下文的函数列表）

DXY-lemon commented 10 months ago

可以支持method级别的fim能力。在"methods_info"中有针对class中具体每个method的描述文本"method_description"和ground truth代码"solution_code"，如果想要做fim，可以给大模型提供待生成method的描述文本，以及class中其他method的代码。此外，数据集中还提供"class_description"，"class_constructor"，以及"import_statement"等上下文依赖信息，可以根据需求给模型提供。

mingwei-liu commented 10 months ago

https://huggingface.co/datasets/FudanSELab/ClassEval, 我们最近在Huggingface也开放了数据集，使用起来更加简单。