R2E (Repository to Environment) is a framework that turns any GitHub repository into an executable environment for evaluating static code generation models and programming agents at scale. It extracts functions and methods from the repository, generates and executes equivalence tests for them using LLMs, and creates an interactive execution environment. These environments can be used to evaluate the quality of LLM generated code.

Installation

1. Install uv to setup R2E.

   curl -LsSf https://astral.sh/uv/install.sh | sh

2. Create a venv, clone, and install.

   git clone https://github.com/r2e-project/r2e.git
   cd r2e && uv venv && source .venv/bin/activate
   uv sync

Key Concept: Equivalence Test Harnesses

R2E introduces Equivalence Tests, a test that checks if two pieces of code are equivalent. In the context of R2E, an Equivalence Test checks if the output of a function/method in a repository is the same as the output of the generated code for the same function/method. Here's an overview of the key principles of R2E:

Find more details and examples at r2e.dev and in our paper.

Usage

R2E provides a convenient CLI to work with. The usual steps are as follows: (1) setup and extract functions from repositories, (2) build and install repositories, and (3) generate and execute Equivalence Tests

Find the complete CLI documentation at CLI.md. Below is a quickstart guide:

1. Setup and Extract

First, choose a unique experiment id (e.g., quickstart) that you can reuse for the entire workflow. Then setup repositories and extract functions from:

r2e setup -r https://github.com/google-research/python-graphs
r2e extract -e quickstart --overwrite_extracted

[!Note] We also support copying from a local path, or processing a list of URLs/local paths from a json file (cli docs).

During extraction all repos cloned into REPOS_DIR are processed. The extracted functions and methods are written to a JSON file. Use --overwrite_extracted to overwrite any existing results.

2. Build and Install

Docker Mode: By default, all repos in REPOS_DIR are installed in a Docker image for sandboxed execution. Find the generated dockerfile in REPOS_DIR. Useful reference: install docker

Local Mode: Use --local which will suggest the steps you need to take to manually to install repos.

r2e build -e quickstart

3. Generate and Execute Tests

R2E provides a single command that runs a series of k generate-execute rounds w/ feedback. The loop continues until min_valid% functions reach a min_cov% branch coverage. Defaults: k=3, min_valid=0.8, and min_cov=0.8.

r2e genexec -e quickstart --save_chat

[!Note] You can also run r2e generate and r2e execute separately (cli docs).

The generated tests are executed in the Docker container. Use --local to execute locally.

Additional Resources

1. Improving Specifications:: Find details about improving the specifications of extracted functions to build benchmarks using LLMs @ Spec Refinement.

2. PAT: Program Analysis Tools: We developed an in-house program and static analysis toolbox that powers R2E along with LLMs. These tools can be used independently too. Learn more about them @ PAT.

Citation

If you use R2E in your research, please cite the following paper:

@inproceedings{
    jain2024r2e,
    title={R2E: Turning any Github Repository into a Programming Agent Environment},
    author={Naman Jain and Manish Shetty and Tianjun Zhang and King Han and Koushik Sen and Ion Stoica},
    booktitle={ICML},
    year={2024},
}