[feature] update workflow based on current input and current result

[Tata0703]

https://github.com/OSU-NLP-Group/SeeAct/blob/main/seeact_package/seeact/demo_utils/inference_engine.py#L189-L207

[Tata0703]

next action generation

[Tata0703]

workflow, ->next step generation, ->action generation->code

[Tata0703]

[{'role': 'system', 'content': 'You are a web search agent designed to perform specific tasks on web pages as instructed by the user. Your primary objectives are:\n\n 1. Execute ONLY the task explicitly provided by the user.\n 2. Perform the task efficiently and accurately using the available functions.\n 3. If there are errors, retry using a different approach within the scope of the given task.\n 4. Once the current task is completed, stop and wait for further instructions.\n\n Critical guidelines:\n - Strictly limit your actions to the current task. Do not attempt additional tasks or next steps.\n - Use only the functions provided to you. Do not attempt to use functions or methods that are not explicitly available.\n - For navigation or interaction with page elements, always use the appropriate bid (browser element ID) when required by a function.\n - Do not try to navigate to external websites or use URLs directly.\n - If a task cannot be completed with the available functions, report the limitation rather than attempting unsupported actions.\n - After completing a task, report its completion and await new instructions. Do not suggest or initiate further actions.\n\n Remember: Your role is to execute the given task precisely as instructed, using only the provided functions and within the confines of the current web page. Do not exceed these boundaries under any circumstances.'}, {'role': 'user', 'content': "In the Sidebar, hover to the right of Docs Hub.\nClick the plus icon.\nIf you don't see Docs Hub, you'll need to pin Docs Hub to the Sidebar."}, {'content': None, 'role': 'assistant', 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"goal":"Hover to the right of Docs Hub in the Sidebar."}', name='navigation'), id='call_dfNYci00KeOix5cxFanJdaQh', type='function')]}, {'tool_call_id': 'call_dfNYci00KeOix5cxFanJdaQh', 'role': 'tool', 'name': 'navigation', 'content': 'The action is: In order to accomplish my goal, I need to hover to the right of Docs Hub in the Sidebar.\n\nhover(\'329\') - the result is: Yes, the goal is accomplished.\n\nBased on the screenshot, the cursor is now hovering to the right of "Docs" in the Sidebar. The blue arrow pointing to the right of the "Docs" link confirms that the action has been performed as intended.'}, {'content': None, 'role': 'assistant', 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"goal":"Click the plus icon."}', name='navigation'), id='call_crYvXyEqKkM9M4Xq3hqgDvBg', type='function')]}, {'tool_call_id': 'call_crYvXyEqKkM9M4Xq3hqgDvBg', 'role': 'tool', 'name': 'navigation', 'content': 'The action is: In order to accomplish my goal, I need to click on the plus icon with bid 427.\n\nclick(\'427\') - the result is: Based on the provided accessibility tree and the screenshot, it appears that the goal has been accomplished. The instruction was to click the plus icon with bid 427 (click(\'427\')), and the visual confirmation is necessary to verify this.\n\nEvaluation:\n1. Accessibility Tree Analysis:\n - Entry [427] button \'\', focused indicates that the button has been focused.\n - There is an alert ([932] paragraph \'\' StaticText \'Do you want to enable browser notifications?\') suggesting the system detected a change and tried to notify about it.\n\n2. Screenshot Analysis:\n - On the left sidebar, the "DZ space" and "List" are highlighted.\n - In the central workspace, a drop-down menu appears with multiple options such as “Rename,” “Copy link,” and “Create new.”\n - This menu also contains other configurations and relevant actions which typically appear after some interactive element (like a button) is clicked.\n\nThese match the navigational context that might follow clicking a plus icon or similar button interaction in an interface where task lists and collaborative editing occur.\n\nHence, considering both the accessibility tree and the visual context, the plus icon with bid 427 has been successfully clicked, and the goal is finished.'}, {'content': 'The task has been completed successfully: I hovered to the right of Docs Hub and clicked the plus icon. Please provide further instructions if needed.', 'role': 'assistant', 'tool_calls': None, 'function_call': None}]

[Tata0703]

[{'role': 'system', 'content': 'You are a web search agent designed to perform specific tasks on web pages as instructed by the user. Your primary objectives are:\n\n 1. Execute ONLY the task explicitly provided by the user.\n 2. Perform the task efficiently and accurately using the available functions.\n 3. If there are errors, retry using a different approach within the scope of the given task.\n 4. Once the current task is completed, stop and wait for further instructions.\n\n Critical guidelines:\n - Strictly limit your actions to the current task. Do not attempt additional tasks or next steps.\n - Use only the functions provided to you. Do not attempt to use functions or methods that are not explicitly available.\n - For navigation or interaction with page elements, always use the appropriate bid (browser element ID) when required by a function.\n - Do not try to navigate to external websites or use URLs directly.\n - If a task cannot be completed with the available functions, report the limitation rather than attempting unsupported actions.\n - After completing a task, report its completion and await new instructions. Do not suggest or initiate further actions.\n\n Remember: Your role is to execute the given task precisely as instructed, using only the provided functions and within the confines of the current web page. Do not exceed these boundaries under any circumstances.'}, {'role': 'user', 'content': "In the Sidebar, hover to the right of Docs Hub.\nClick the plus icon.\nIf you don't see Docs Hub, you'll need to pin Docs Hub to the Sidebar."}, {'content': None, 'role': 'assistant', 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"goal":"Hover to the right of Docs Hub in the Sidebar and click the plus icon."}', name='navigation'), id='call_s8kf4zyLXosqL9F6nYIlfwz5', type='function')]}, {'tool_call_id': 'call_s8kf4zyLXosqL9F6nYIlfwz5', 'role': 'tool', 'name': 'navigation', 'content': 'The action is: In order to accomplish the goal of hovering to the right of "Docs Hub" in the Sidebar and clicking the plus icon, I will first hover over the button next to "Docs" in the Sidebar.\n\nhover(\'345\') - the result is: The goal of hovering to the right of "Docs Hub" in the Sidebar and clicking the plus icon is not yet complete. \n\nBased on the screenshot, we can see that the mouse pointer is currently hovering over the plus icon next to "Docs" in the Sidebar. However, it does not appear that the plus icon has been clicked yet. The visual feedback, such as an expanded menu or any subsequent action after clicking the plus icon, is not present in the screenshot.\n\nTherefore, the goal requires the additional step of clicking the plus icon to finish the task.'}, {'content': None, 'role': 'assistant', 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"goal":"Click the plus icon next to Docs Hub in the Sidebar."}', name='navigation'), id='call_6HaC7jLMtyXmNWBbwp7DVrnx', type='function')]}, {'tool_call_id': 'call_6HaC7jLMtyXmNWBbwp7DVrnx', 'role': 'tool', 'name': 'navigation', 'content': 'The action is: In order to accomplish the goal, I need to click the plus icon next to "Docs Hub" in the Sidebar, which seems to be the button with bid 329.\n\nclick(\'329\') - the result is: The goal is to click the plus icon next to "Docs Hub" in the Sidebar. From the Accessibility Tree information provided and the screenshot, it is clear that the plus icon next to "Docs Hub" has been clicked because the UI state includes an expanded view and shows various options related to documents and tasks.\n\nBased on the Accessibility Tree, we see that the interface\'s state includes detailed document and task-related controls (such as "Add Task," "Summarize list," "Project Update," etc.), which are typical results from expanding a documents hub or related menu.\n\nThus, the goal of clicking the plus icon next to "Docs Hub" in the Sidebar has been accomplished.'}, {'content': 'The task of hovering to the right of "Docs Hub" in the Sidebar and clicking the plus icon has been successfully completed. Please provide further instructions if needed.', 'role': 'assistant', 'tool_calls': None, 'function_call': None}]

[Tata0703]

[{'role': 'system', 'content': 'You are a web search agent designed to perform specific tasks on web pages as instructed by the user. Your primary objectives are:\n\n 1. Execute ONLY the task explicitly provided by the user.\n 2. Perform the task efficiently and accurately using the available functions.\n 3. If there are errors, retry using a different approach within the scope of the given task.\n 4. Once the current task is completed, stop and wait for further instructions.\n\n Critical guidelines:\n - Strictly limit your actions to the current task. Do not attempt additional tasks or next steps.\n - Use only the functions provided to you. Do not attempt to use functions or methods that are not explicitly available.\n - For navigation or interaction with page elements, always use the appropriate bid (browser element ID) when required by a function.\n - Do not try to navigate to external websites or use URLs directly.\n - If a task cannot be completed with the available functions, report the limitation rather than attempting unsupported actions.\n - After completing a task, report its completion and await new instructions. Do not suggest or initiate further actions.\n\n Remember: Your role is to execute the given task precisely as instructed, using only the provided functions and within the confines of the current web page. Do not exceed these boundaries under any circumstances.'}, {'role': 'user', 'content': "In the Sidebar, hover to the right of Docs Hub.\nClick the plus icon.\nIf you don't see Docs Hub, you'll need to pin Docs Hub to the Sidebar."}, {'content': None, 'role': 'assistant', 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"goal":"Hover to the right of Docs Hub in the Sidebar, then click the plus icon."}', name='navigation'), id='call_LM8DlBF2ysYO1HASfegGgtVe', type='function')]}, {'tool_call_id': 'call_LM8DlBF2ysYO1HASfegGgtVe', 'role': 'tool', 'name': 'navigation', 'content': 'The action is: To accomplish the goal of hovering to the right of "Docs Hub" in the Sidebar and clicking the plus icon, I will first hover over "Docs Hub."\n\nhover(\'329\') - the result is: Yes, the goal is accomplished. \n\nThe original goal was to hover to the right of "Docs Hub" in the Sidebar and click the plus icon. The Accessibility Tree shows the hover action and updated elements. In the screenshot, the sidebar and the elements around the "Docs Hub" and adjacent buttons are visible. The pointer is hovering in the correct area, and you can see the Sidebar, including the "Docs" link visible at access key [329], which aligns with the section you were targeting. \n\nThus, the action of hovering and clicking the plus icon next to "Docs Hub" has been achieved as intended.'}, {'content': None, 'role': 'assistant', 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"goal":"Click the plus icon next to Docs Hub."}', name='navigation'), id='call_qLe8dCWajMIDOiool4EnWNMm', type='function')]}, {'tool_call_id': 'call_qLe8dCWajMIDOiool4EnWNMm', 'role': 'tool', 'name': 'navigation', 'content': 'The action is: In order to accomplish my goal, I need to click on the plus icon next to "Docs Hub". This appears to be the button with bid \'326\'.\n\nclick(\'326\') - the result is: Yes, the goal is finished. You needed to click the plus icon next to "Docs Hub," which corresponds to the button with bid \'326\'. The updated accessibility tree reflects that the button \'Create Doc\' with bid \'326\' is now focused, demonstrating that the click action was executed successfully:\n\n\n[326] button \'Create Doc\', focused\n\n\nAdditionally, the visual elements in the screenshot support this, showing the \'Create Doc\' button in the state of being clicked or focused. This confirms that the required action has been completed.'}, {'content': 'The tasks have been completed successfully: \n\n1. Hovered to the right of "Docs Hub" in the Sidebar.\n2. Clicked the plus icon next to "Docs Hub."\n\nIf you have further instructions or another task, please let me know!', 'role': 'assistant', 'tool_calls': None, 'function_call': None}]

[Tata0703]

`[{'role': 'system', 'content': 'You are a web search agent designed to perform specific tasks on web pages as instructed by the user. Your primary objectives are:\n\n 1. Execute ONLY the task explicitly provided by the user.\n 2. Perform the task efficiently and accurately using the available functions.\n 3. If there are errors, retry using a different approach within the scope of the given task.\n 4. Once the current task is completed, stop and wait for further instructions.\n\n Critical guidelines:\n - Strictly limit your actions to the current task. Do not attempt additional tasks or next steps.\n - Use only the functions provided to you. Do not attempt to use functions or methods that are not explicitly available.\n - For navigation or interaction with page elements, always use the appropriate bid (browser element ID) when required by a function.\n - Do not try to navigate to external websites or use URLs directly.\n - If a task cannot be completed with the available functions, report the limitation rather than attempting unsupported actions.\n - After completing a task, report its completion and await new instructions. Do not suggest or initiate further actions.\n\n Remember: Your role is to execute the given task precisely as instructed, using only the provided functions and within the confines of the current web page. Do not exceed these boundaries under any circumstances.'},

{'role': 'user', 'content': "In the Sidebar, hover to the right of Docs Hub.\nClick the plus icon.\nIf you don't see Docs Hub, you'll need to pin Docs Hub to the Sidebar."},

{'content': None, 'role': 'assistant', 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"goal":"Hover to the right of Docs Hub in the Sidebar."}', name='navigation'), id='call_WimuwbW4xDTB7ZmcvijKQ5ce', type='function')]}, {'tool_call_id': 'call_WimuwbW4xDTB7ZmcvijKQ5ce', 'role': 'tool', 'name': 'navigation', 'content': 'The action is: hover(\'334\') - the result is: Based on the screenshot and the updated Accessibility Tree, the goal of hovering to the right of "Docs Hub" in the Sidebar ("hover(\'334\')") appears to be finished.\n\nFrom the Accessibility Tree:\n- Node [334] represents the "Docs" link.\n- The sibling elements and nodes around [334] are visible, demonstrating that the focus or operation around this element has occurred.\n\nFrom the Screenshot:\n- The blue arrow (indicative of a cursor hover action) is pointing towards the "Docs" button in the sidebar.\n- This confirms visually that the hover action was indeed performed to the right of the "Docs" section.\n\nHence, both the visual element in the screenshot (the blue arrow pointing toward "Docs") and the corresponding updated element in the Accessibility Tree ([334] link \'Docs\') indicate that the original goal is accomplished.'},

{'role': 'user', 'content': "\n Current plan: In the Sidebar, hover to the right of Docs Hub.\nClick the plus icon.\nIf you don't see Docs Hub, you'll need to pin Docs Hub to the Sidebar.\n \n Based on the progress made so far, please provide:\n \n 1. An updated complete plan\n 2. A list of tasks that have already been completed\n 3. The next task to be tackled\n 4. An explanation of changes and their rationale\n \n Format your response as follows:\n \n Updated Plan:\n - [Step 1]\n - [Step 2]\n - ...\n \n Completed Tasks:\n - [Task 1]\n - [Task 2]\n - ...\n \n Next Task:\n [Specify the next task to be done]\n "},

{'content': "Updated Plan:\n- Hover to the right of Docs Hub in the Sidebar.\n- Click the plus icon.\n- If you don't see Docs Hub, pin Docs Hub to the Sidebar.\n\nCompleted Tasks:\n- Hovered to the right of Docs Hub in the Sidebar.\n\nNext Task:\nClick the plus icon.", 'role': 'assistant', 'tool_calls': None, 'function_call': None},

{'content': None, 'role': 'assistant', 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"goal":"Click the plus icon."}', name='navigation'), id='call_sLQNFFVzyZzPZPeZYkbSjoAA', type='function')]}, {'tool_call_id': 'call_sLQNFFVzyZzPZPeZYkbSjoAA', 'role': 'tool', 'name': 'navigation', 'content': 'The action is: In order to accomplish the goal of clicking the plus icon, I need to click the button with bid 563.\n\nclick(\'563\') - the result is: Yes, the goal of clicking the plus icon appears to be finished. \n\nThe relevant section of the Accessibility Tree has been updated to reflect that the button with the bid 563 has been clicked. In the screenshot, you can see that the button 562 is indicated as expanded (expanded=True), which is likely the result of the action performed.\n\nAdditionally, the visual change in the screenshot supports this action, showing that more elements (likely tasks or items) are now displayed or editable, which is the expected outcome of clicking a plus or "add" button.\n\nSo, based on the provided information and the visual evidence in the screenshot, the goal of clicking the plus icon has been successfully accomplished.'}, {'role': 'user', 'content': "\n Current plan: Updated Plan:\n- Hover to the right of Docs Hub in the Sidebar.\n- Click the plus icon.\n- If you don't see Docs Hub, pin Docs Hub to the Sidebar.\n\nCompleted Tasks:\n- Hovered to the right of Docs Hub in the Sidebar.\n\nNext Task:\nClick the plus icon.\n \n Based on the progress made so far, please provide:\n \n 1. An updated complete plan\n 2. A list of tasks that have already been completed\n 3. The next task to be tackled\n 4. An explanation of changes and their rationale\n \n Format your response as follows:\n \n Updated Plan:\n - [Step 1]\n - [Step 2]\n - ...\n \n Completed Tasks:\n - [Task 1]\n - [Task 2]\n - ...\n \n Next Task:\n [Specify the next task to be done]\n "}, {'content': "Updated Plan:\n- Hover to the right of Docs Hub in the Sidebar.\n- Click the plus icon.\n- If you don't see Docs Hub, pin Docs Hub to the Sidebar.\n\nCompleted Tasks:\n- Hovered to the right of Docs Hub in the Sidebar.\n- Clicked the plus icon.\n\nNext Task:\nIf you don't see Docs Hub, pin Docs Hub to the Sidebar.", 'role': 'assistant', 'tool_calls': None, 'function_call': None}, {'content': None, 'role': 'assistant', 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"goal":"Check if Docs Hub is visible in the Sidebar."}', name='navigation'), id='call_oveh8jCysnO41eIg36yq6wFI', type='function')]}, {'tool_call_id': 'call_oveh8jCysnO41eIg36yq6wFI', 'role': 'tool', 'name': 'navigation', 'content': 'The "Docs Hub" is represented with the link \'Docs\' with bid [334] in the accessibility tree, and its corresponding button [328] for the settings.\n\nclick(\'334\') - Task failed with action error: TimeoutError: Timeout 500ms exceeded.\n=========================== logs ===========================\nwaiting for get_by_test_id("334")\n locator resolved to <a data-unique-test-id="334" browsergym_set_of_mark…>…\nattempting click action\n waiting for element to be visible, enabled and stable\n element is visible, enabled and stable\n scrolling into view if needed\n done scrolling\n <div data-unique-test-id="3267" browsergym_set_of_mar…>

[Tata0703]

https://github.com/PathOnAI/LiteWebAgent/blob/main/litewebagent/agents/HighLevelPlanningAgent.py

[Tata0703]

https://arxiv.org/abs/2305.04091