In this work, we describe a newly forming research area Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways.
We present a simple yet effective ADAS algorithm named Meta Agent Search to demonstrate that agents can invent novel and powerful agent designs. In Meta Agent Search, a "meta" agent iteratively programs interesting new agents in code based on previous discoveries.
Setup
conda create -n adas python=3.11
conda activate adas
pip install -r requirements.txt
# provide your OpenAI API key
export OPENAI_API_KEY="YOUR KEY HERE"
Running Instructions
Running Meta Agent Search
To run experiments for each domain, navigate to its respective folder. The code in each folder is self-contained. Launch experiments using the search.py script located in each domain's folder.
python {DOMAIN}/search.py
Replace {DOMAIN} with the specific domain folder name {_arc, _drop, _mgsm, ...} to run the experiment for.
Customizing Meta Agent Search for New Domains
You can easily adapt the code to search for new domains. To do so, follow these steps:
Modify the evaluate_forward_fn() function and adjust any necessary formatting prompts (e.g. this line) in the search.py file.
Consider adding additional basic functions for the meta agent to utilize during the design process (similar to this line).
Update the domain-specific information within the prompts to match the requirements of your new domain (e.g. this line).
Run the search and evaluation on your new domain.
Safety Consideration
[!WARNING]
The code in this repository involves executing untrusted model-generated code. We strongly advise users to be aware of this safety concern. While it is highly unlikely that model-generated code will perform overtly malicious actions in our current settings and with the models we use, such code may still act destructively due to limitations in model capability or alignment. By using this repository, you acknowledge and accept these risks.
Citing
If you find this project useful, please consider citing:
@article{hu2024ADAS,
title={Automated Design of Agentic Systems},
author={Hu, Shengran and Lu, Cong and Clune, Jeff},
journal={arXiv preprint arXiv:2408.08435},
year={2024}
}
ShellLM
commented
3 weeks ago
Automated Design of Agentic Systems
In this work, we describe a newly forming research area Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways.
We present a simple yet effective ADAS algorithm named Meta Agent Search to demonstrate that agents can invent novel and powerful agent designs. In Meta Agent Search, a "meta" agent iteratively programs interesting new agents in code based on previous discoveries.
Setup
Running Instructions
Running Meta Agent Search
To run experiments for each domain, navigate to its respective folder. The code in each folder is self-contained. Launch experiments using the
search.pyscript located in each domain's folder.Replace
{DOMAIN}with the specific domain folder name {_arc,_drop,_mgsm, ...} to run the experiment for.Customizing Meta Agent Search for New Domains
You can easily adapt the code to search for new domains. To do so, follow these steps:
Modify the
evaluate_forward_fn()function and adjust any necessary formatting prompts (e.g. this line) in thesearch.pyfile.Consider adding additional basic functions for the meta agent to utilize during the design process (similar to this line).
Update the domain-specific information within the prompts to match the requirements of your new domain (e.g. this line).
Run the search and evaluation on your new domain.
Safety Consideration
Citing
If you find this project useful, please consider citing:
Suggested labels
None