😽News | 🐈Setup | 🙀Localization | 😼Repair | 🧶Comparison | 🐈⬛Artifacts | 📝Citation | 😻Acknowledgement
Agentless is an agentless approach to automatically solve software development problems. To solve each issue, Agentless follows a simple two phase process: localization and repair.
First create the environment
git clone https://github.com/OpenAutoCoder/Agentless.git
cd Agentless
conda create -n agentless python=3.11
conda activate agentless
pip install -r requirements.txt
export PYTHONPATH=$PYTHONPATH:$(pwd)
Then export your OpenAI API key
export OPENAI_API_KEY={key_here}
Now you are ready to run Agentless on the problems in SWE-bench! We now go through a step-by-step example of how to run Agentless.
[!NOTE]
To reproduce the full SWE-bench lite experiments and follow our exact setup as described in the paper. Please see this README
[!TIP]
For localization, you can use
--target_id
to specific a particular bug you want to target.For example
--target_id=django__django-11039
In localization, the goal is find the locations in source code where we need to edit to fix the issues. Agentless uses a 3-stage localization step to first localize to specific files, then to relevant code elements, and finally to fine-grained edit locations.
[!TIP]
Since for each issue in the benchmark we need to checkout the repository and process the files, you might want to save some time by downloading the preprocessed data here: swebench_lite_repo_structure.zip
After downloading, please unzip and export the location as such
export PROJECT_FILE_LOC={folder which you saved}
Run the following command to generate the edit locations:
mkdir results # where we will save our results
python agentless/fl/localize.py --file_level --related_level --fine_grain_line_level \
--output_folder results/location --top_n 3 \
--compress \
--context_window=10
This will save all the localized locations in results/location/loc_outputs.jsonl
with the logs saved in results/location/localize.log
Using the edit locations (i.e., found_edit_locs
) from before, we now perform repair.
Agentless generates multiple patches per issue (controllable via parameters) and then perform majority voting to select the final patch for submission
Run the following command to generate the patches:
python agentless/repair/repair.py --loc_file results/location/loc_outputs.jsonl \
--output_folder results/repair \
--loc_interval --top_n=3 --context_window=10 \
--max_samples 10 --cot --diff_format \
--gen_and_process
This command generates 10 samples (1 greedy and 9 via temperature sampling) as defined --max_samples 10
. The --context_window
indicates the amount of code lines before and after each localized edit location we provide to the model for repair. The repair results is saved in results/repair/output.jsonl
, which contains the raw output of each sample as well as the any trajectory information (e.g., number of tokens). The complete logs are also saved in results/repair/repair.log
[!NOTE]
We also perform post-processing to generate the complete git-diff patch for each repair samples.
You can find the individual patch in
results/repair/output_{i}_processed.jsonl
wherei
is the sample number.
Finally, we perform majority voting to select the final patch to solve each issue. Run the following command:
python agentless/repair/rerank.py --patch_folder results/repair --num_samples 10 --deduplicate --plausible
In this case, we use --num_samples 10
to pick from the 10 samples we generated previously, --deduplicate
to apply normalization to each patch for better voting, and --plausible
to select patches that can pass the previous regression tests (warning: this feature is not yet implemented)
This command will produced the all_preds.jsonl
that contains the final selected patch for each instance_id which you can then directly use your favorite way of testing SWE-bench for evaluation!
Below shows the comparison graph between Agentless and the best open-source agent-based approaches on SWE-bench lite
You can download the complete artifacts of Agentless in our v0.1.0 release:
You can also checkout classification/
folder to obtain our manual classifications of SWE-bench-lite as well as our filtered SWE-bench-lite-S problems.
@article{agentless,
author = {Xia, Chunqiu Steven and Deng, Yinlin and Dunn, Soren and Zhang, Lingming},
title = {Agentless: Demystifying LLM-based Software Engineering Agents},
year = {2024},
journal = {arXiv preprint},
}
[!NOTE]
The first two authors contributed equally to this work, with author order determined via Nigiri