xhluca
commented
2 hours ago One example, agent-xray fails when loading pickle stored during experiment from a ResponsiveAgentArgs (a custom agent arg). Only way to solve this is to copy the code of agent-xray and add the import myself.
AttributeError: Can't get attribute 'ResponsiveAgentArgs' on <module '__main__' from '/home/AgentLab/src/agentlab/analyze/agent_xray.py'>. Did you mean: '_exp_args'?
Traceback (most recent call last):
File "/home/AgentLab/venv/lib/python3.10/site-packages/gradio/queueing.py", line 536, in process_events
response = await route_utils.call_process_api(
File "/home/AgentLab/venv/lib/python3.10/site-packages/gradio/route_utils.py", line 322, in call_process_api
output = await app.get_blocks().process_api(
File "/home/AgentLab/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1935, in process_api
result = await self.call_function(
File "/home/AgentLab/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1520, in call_function
prediction = await anyio.to_thread.run_sync( # type: ignore
File "/home/AgentLab/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/home/AgentLab/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
return await future
File "/home/AgentLab/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 943, in run
result = context.run(func, *args)
File "/home/AgentLab/venv/lib/python3.10/site-packages/gradio/utils.py", line 826, in wrapper
response = f(*args, **kwargs)
File "/home/AgentLab/src/agentlab/analyze/agent_xray.py", line 924, in new_exp_dir
info.result_df = inspect_results.load_result_df(info.exp_list_dir, progress_fn=progress.tqdm)
File "/home/AgentLab/src/agentlab/analyze/inspect_results.py", line 159, in load_result_df
df = pd.DataFrame([exp_result.get_exp_record() for exp_result in result_list])
File "/home/AgentLab/src/agentlab/analyze/inspect_results.py", line 159, in <listcomp>
df = pd.DataFrame([exp_result.get_exp_record() for exp_result in result_list])
File "/home/AgentLab/venv/lib/python3.10/site-packages/browsergym/experiments/loop.py", line 671, in get_exp_record
record.update(self.flat_exp_args)
File "/home/AgentLab/venv/lib/python3.10/site-packages/browsergym/experiments/loop.py", line 662, in flat_exp_args
exp_args = asdict(self.exp_args)
File "/home/AgentLab/venv/lib/python3.10/site-packages/browsergym/experiments/loop.py", line 574, in exp_args
self._exp_args = pickle.load(f)
AttributeError: Can't get attribute 'ResponsiveAgentArgs' on <module '__main__' from '/home/AgentLab/src/agentlab/analyze/agent_xray.py'>. Did you mean: '_exp_args'?
Consider this pickle usage: https://github.com/ServiceNow/BrowserGym/blob/fd68d2fe43ffaa49b3d2905305a6e8bb4c84eb87/browsergym/experiments/src/browsergym/experiments/loop.py#L595
If a custom args, which was not defined in browsergym, is used for constructing the agent, then this pickle load will fail. One instance is in agent-xray, where the steps are stored in the pickle format.
Not sure what's the best way to address this, but it's something to consider when using custom agents that are not the genericagents or genericagentargs shipped with agentlab. Another consideration is security. I think it's worth consdering this after deadline.