search

ShengranHu / ADAS

Automated Design of Agentic Systems
Apache License 2.0
1k stars 148 forks source link

agentSystem.forward is not working in MMLU task #10

Open kwangje opened 2 months ago

kwangje commented 2 months ago

I'm having an issue where the content of the results is coming out empty. Here is a snippet of the code and result:

The result for task_queue[99] is:

Info(name='task', author='User', content='Answer the following multiple choice question.\n\nWhen a fusion reaction converts a pair of hydrogen isotopes to an alpha particle and a neutron most of the energy released is in the form of\n\n(A) gamma radiation.\n(B) kinetic energy of the alpha particle.\n(C) kinetic energy of the neutron.\n(D) All of these about equally.', iteration_idx=-1)
agentSystem = AgentSystem()

acc_list = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
    results = list(tqdm(executor.map(agentSystem.forward, task_queue), total=len(task_queue)))

The result for results[0] is:

Info(name='final_answer', author='Self-Critique Agent', content=Info(name='answer', author='Answer Generator y5Cv', content='', iteration_idx=-1), iteration_idx=-1)

The results of the code always show that the content is empty. I am not sure why this is happening.

Could it be that the issue arises because the code implementations for MMLU and other tasks are missing, whereas the AgentSystem class works fine for ARC tasks?

I would appreciate any help you can provide.

ShengranHu commented 2 months ago

Hello. Could you provide more context on when this error occurred? If it only happens occasionally during the evaluation of generated agents, this is expected, as generated agents often have bugs.​