CodeApex is a bilingual programming evaluation benchmark for Large Language Models. It consists two basic programming tasks: programming comprehension and code generation. Programming Comprehension Test consists of 250 multiple choice quesitions, including conceptual understanding, commonsense reasoning, and multi-hop reasoning three question categories. Code generation Task consists of 476 C++ based algorithm problems, covering common algorithm knowledge points like binary search, depth-firsts-search and so on. In the future, CodeApex will publish other code-related functional tests, such as code correction. This is an evaluation repository for the Paper "CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models".

News

• [2023.09.05] CodeApex is published on arXiv and Github. Benchmark website is open for commission now.

Table of Contents

• Leaderboard
• Data
• How to Evaluation on CodeApex
• How to Submit
• Licenses
• Citation

Leaderboard

The public leaderboard is presented in Leaderboard.

Programming Comprehension

Code Generation

Data

Test data are published in this repo.

First, clone the this repo:

git clone https://github.com/SJTU-LIT/ceval.git

The data is in ProgrammingComprehension/testcases and CodeGeneration/data, whose format is json.

The format of programming comprehension is:

[
    {
        "question": "If there is a definition: char str[] = {'h','1','2','0','a','b'}; const char *p = str; Which of the following statements is correct:____\n",
        "A": "p[2] = 's'",
        "B": "strcpy(str,\"123456\")",
        "C": "strcpy(p,\"abc\")",
        "D": "strcpy(str,\"abc\")",
        "category": 1,
        "id": 0
    },
    ...
]

The format of code generation is:

[
   {
        "id": 1964,
        "problem_description": "Given 4 positive integers a, b, c, d, calculate the value of the expression (a*b*c)%d, where % represents the modulo operation.",
        "function_declaration": "int calculate_remainder(int a, int b, int c, int d)",
        "code_context": "#include<cstdio>\n\n// function start\n\n// function end\n\nint main(){\n    int a,b,c,d;\n    scanf(\"%d%d%d%d\",&a,&b,&c,&d);\n    // calling start\n    int result = calculate_remainder(a, b, c, d);\n    // calling end\n    printf(\"%d\", result);\n}",
        "example": "[{\"input\": \"2 3 4 5\", \"output\": \"4\"}]",
        "time_limit": 1000,
        "memory_limit": 256
    },
    ...
]

How to Evaluation on CodeApex

Programming Comprehension

You can evaluate your model's response on our website. Users should be responsible for the correctness and compliance of their inputs. The format of answer generated by LLM is json, and the json file is divided into three dictionaries in order, representing the answers for CU, CR, and MCR, with each dictionary's answers sorted by ID within the dictionary. We provide an example example.json. Your input should be a npy file containing your answer to the testcases, and run the deal_answer.py to generate the json file for evaluation.

How to submit?

Code Generation

Online evaluation of code generation tasks is being set up. Coming soon.

Citation

Please cite using the following bibtex entry:

@misc{fu2023codeapex,
      title={CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models}, 
      author = {Fu, Lingyue and Chai, Huacan and Luo, Shuang and Du, Kounianhua and Zhang, Weiming and Fan, Longteng and Lei, Jiayi and Rui, Renting and Lin, Jianghao and Fang, Yuchen and Liu, Yifan and Wang, Jingkuan and Qi, Siyuan and Zhang, Kangning and Zhang, Weinan and Yu, Yong}, 
      year={2023},
      eprint={2309.01940},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}