Error while running with local llm server

[anupamme]

I'm using https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-8B-GGUF (Q4) in LM studio.

When I run this command:

python -m scripts.evaluate.debug_webarena

I get following exception in the main terminal:

Next Action: type [search_box_id] [CVS UPMC Shadyside] [1] This action types the query "CVS UPMC Shadyside" into the search box on the webpage, which will help the web agent find the closest location. Since we want to measure the distance between the closet CVS and UPMC Shadyside by walking, this is an appropriate step in the process.

Note:

Replace search_box_id with the actual id of the search box if available. If not, you can try using a generic selector for the search input field like "input[type='search']". Also note that the press_enter_after parameter is set to 1 which means that the 'Enter' key will

OBJECTIVE: Measure distance between CVS (closet one) and UPMC Shadyside by walking

URL: http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108

PREVIOUS HISTORY: []

ACTION: type [search_box_id] [CVS UPMC Shadyside] [1] This action types the query "CVS UPMC Shadyside" into the search box on the webpage, which will help the web agent find the closest location. Since we want to measure the distance between the closet CVS and UPMC Shadyside by walking, this is an appropriate step in the process.

Note:

Replace search_box_id with the actual id of the search box if available. If not, you can try using a generic selector for the search input field like "input[type='search']". Also note that the press_enter_after parameter is set to 1 which means that the 'Enter' key will [Step 1] type [search_box_id] [CVS UPMC Shadyside] [1] This action types the query "CVS UPMC Shadyside" into the search box on the webpage, which will help the web agent find the closest location. Since we want to measure the distance between the closet CVS and UPMC Shadyside by walking, this is an appropriate step in the process.

Note:

Replace search_box_id with the actual id of the search box if available. If not, you can try using a generic selector for the search input field like "input[type='search']". Also note that the press_enter_after parameter is set to 1 which means that the 'Enter' key will Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/Users/mediratta/code/ml/step_dspy/scripts/evaluate/debug_webarena.py", line 44, in run() File "/Users/mediratta/code/ml/step_dspy/scripts/evaluate/debug_webarena.py", line 39, in run status = agent.act(objective=objective, env=env) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mediratta/code/ml/step_dspy/lib/agents/agent.py", line 51, in act status = env.step(action) ^^^^^^^^^^^^^^^^ File "/Users/mediratta/code/ml/step_dspy/lib/environments/webarena.py", line 111, in step action_cmd = self.call_right_action(action) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mediratta/code/ml/step_dspy/lib/environments/webarena.py", line 161, in call_right_action return create_id_based_action(action_str) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<@beartype(browser_env.actions.create_id_based_action) at 0x10b057f60>", line 32, in create_id_based_action File "/Users/mediratta/code/ml/step_dspy/webarena/browser_env/actions.py", line 1475, in create_id_based_action raise ActionParsingError(f"Invalid type action {action_str}") browser_env.actions.ActionParsingError: Invalid type action type [search_box_id] [CVS UPMC Shadyside] [1] This action types the query "CVS UPMC Shadyside" into the search box on the webpage, which will help the web agent find the closest location. Since we want to measure the distance between the closet CVS and UPMC Shadyside by walking, this is an appropriate step in the process.

Note:

Replace search_box_id with the actual id of the search box if available. If not, you can try using a generic selector for the search input field like "input[type='search']". Also note that the press_enter_after parameter is set to 1 which means that the 'Enter' key will [1]

The console on the LM studio server looks ok (no error).

What is the possible reason here?

[viig99]

the action should be limited to only the particular line, but sometimes these model add extra stuff like reasoning into the action line, either: 1) we can parse the response and check / grab the line which is for the action and feed that to the parser 2) get the model to only output the line by some means

I think 1 is easier than 2, if you want to take it up.

[anupamme]

I can take it up but need a bit more help.