search

OpenAdaptAI / OpenAdapt

Open Source Generative Process Automation (i.e. Generative RPA). AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models
https://www.OpenAdapt.AI
MIT License
992 stars 138 forks source link

[Bug]: invalid action produced by VisualReplayStrategy #654

Open abrichr opened 6 months ago

abrichr commented 6 months ago

Describe the bug

2024-05-06 14:45:05.598 | INFO     | openadapt.adapters.openai:prompt:177 - result=                                                                                   
('```json\n'                                                                                                                                                          
 '{\n'                                                                                                                                                                
 '  "actions": [\n'                                                                                                                                                   
 '    {"name": "type", "text": "<cmd>-<space>", "canonical_text": '                                                                                                   
 '"<cmd>-<49>"},\n'                                                                                                                                                   
 '    {"name": "type", "text": "c-a-l-c-u-l-a-t-o-r", "canonical_text": '                                                                                             
 '"c-a-l-c-u-l-a-t-o-r"},\n'                                                                                                                                          
 '    {"name": "type", "text": "<enter>", "canonical_text": "<36>"},\n'                                                                                               
 '    {"name": "singleclick", "active_segment_description": "(None)", '                                                                                               
 '"mouse_button_name": "left"},\n'                                                                                                                                    
 '    {"name": "singleclick", "active_segment_description": "Calculator number '                                                                                      
 '2 button", "mouse_button_name": "left"},\n'                                                                                                                         
 '    {"name": "singleclick", "active_segment_description": "Calculator '                                                                                             
 'multiplication button", "mouse_button_name": "left"},\n'                                                                                                            
 '    {"name": "singleclick", "active_segment_description": "Calculator number '                                                                                      
 '3 button", "mouse_button_name": "left"},\n'                                                                                                                         
 '    {"name": "singleclick", "active_segment_description": "Calculator '                                                                                             
 'addition button", "mouse_button_name": "left"},\n'                                                                                                                  
 '    {"name": "singleclick", "active_segment_description": "Calculator number '                                                                                      
 '4 button", "mouse_button_name": "left"},\n'                                                                                                                         
 '    {"name": "singleclick", "active_segment_description": "Calculator '                                                                                             
 'subtraction button", "mouse_button_name": "left"},\n'                                                                                                               
 '    {"name": "singleclick", "active_segment_description": "Calculator number '                                                                                      
 '5 button", "mouse_button_name": "left"},\n'                                                                                                                         
 '    {"name": "singleclick", "active_segment_description": "Calculator equals '                                                                                      
 'button", "mouse_button_name": "left"},\n'                                                                                                                           
 '    {"name": "singleclick", "active_segment_description": "Calculator screen '                                                                                       'with minimization, maximization, and closing buttons", "mouse_button_name": '                                                                                       
 '"left"},\n'                                                                                                                                                         
 '    {"name": "type", "text": "<cmd>-<tab>", "canonical_text": "<cmd>-<48>"}\n'                                                                                      
 '  ]\n'                                                                                                                                                              
 '}\n'                                                                                                                                                                
 '```')
...
2024-05-06 14:45:16.264 | INFO     | openadapt.strategies.visual:find_similar_image_segmentation:434 - similarity_index=0.980340101109988                             
2024-05-06 14:45:16.265 | WARNING  | openadapt.strategies.visual:get_next_action_event:275 - exc=ValueError("'(None)' is not in list")                                2024-05-06 14:45:16.282 | INFO     | openadapt.strategies.visual:find_similar_image_segmentation:434 - similarity_index=0.980340101109988                             
2024-05-06 14:45:16.283 | WARNING  | openadapt.strategies.visual:get_next_action_event:275 - exc=ValueError("'(None)' is not in list")                                
2024-05-06 14:45:16.301 | INFO     | openadapt.strategies.visual:find_similar_image_segmentation:434 - similarity_index=0.980340101109988                             
2024-05-06 14:45:16.302 | WARNING  | openadapt.strategies.visual:get_next_action_event:275 - exc=ValueError("'(None)' is not in list")                                
2024-05-06 14:45:16.317 | INFO     | openadapt.strategies.visual:find_similar_image_segmentation:434 - similarity_index=0.980340101109988                             
2024-05-06 14:45:16.317 | WARNING  | openadapt.strategies.visual:get_next_action_event:275 - exc=ValueError("'(None)' is not in list") 
...

To Reproduce

python -m openadapt.record "using the calculator" cmd+space -> calculator -> enter -> 5 x 6 = -> cmd+w -> ctrl+c python -m openadapt.replay VisualReplayStrategy --instructions ""calculate 2*3+4-5"