[Bug]: model generated JSON snippet exceeds output context length

[abrichr]

Describe the bug

 python -m openadapt.replay SegmentReplayStrategy "replay verbatim"

This happens when there are too many segments due to a lack of filtering, e.g.

('```json\n'                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      14:20:57 [36/95455]
 '{\n'                                               
 '  "descriptions": [\n'                                     
 '    [1, "Calculator GUI"],\n'                              
 '    [2, "Dark grey background area"],\n'                   
 '    [3, "Dark grey background area with text"],\n' 
 '    [4, "Dark grey background area"],\n'           
 '    [5, "Dark grey background area with text"],\n' 
 '    [6, "Button with \'7\'"],\n'                    
 '    [7, "Round button"],\n'                                
 '    [8, "Dark grey background area with text"],\n'         
 '    [9, "Buttons with \'2\' and \'3\'"],\n'
 '    [10, "Dark grey background area with text"],\n'
 '    [11, "Dark grey background area with text"],\n'        
 '    [12, "Dark grey background area with a line"],\n'
 '    [13, "Dark grey background area with a line"],\n'                                                                                                                                                                                                   
 '    [14, "Button with icon or text"],\n'           
 '    [15, "Button with \'8\'"],\n'                  
 '    [16, "Button with \'-\'"],\n'           
 '    [17, "Dark grey background area"],\n'           
 '    [18, "Dark grey background area with text"],\n'       
 '    [19, "Dark grey background area"],\n'       
 '    [20, "Dark grey background area"],\n'
 '    [21, "Button with icon or text"],\n'           
 '    [22, "Round button"],\n'             
 '    [23, "Dark grey background area with text"],\n'
 '    [24, "Dark grey background area with text"],\n'
 '    [25, "Dark grey background area with text"],\n'
 '    [26, "Round button"],\n'                      
 '    [27, "Button with text"],\n'                   
 '    [28, "Dark grey background area with text"],\n'
 '    [29, "Button with \'1\'"],\n'                          
 '    [30, "Dark grey background area"],\n'   
 '    [31, "Button with text"],\n'         
 '    [32, "Dark grey background area"],\n'          
 '    [33, "Round button"],\n'                       
 '    [34, "Dark grey background area with text"],\n'
 '    [35, "Button with \'C\'"],\n'                  
 '    [36, "Dark grey background area"],\n'                  
 '    [37, "Dark grey background area with text"],\n'
 '    [38, "Text or icon on a dark grey background"],\n'
 '    [39, "Dark grey area with text"],\n'
 '    [40, "Dark grey background area"],\n'                  
 '    [41, "Column of buttons with icons"],\n'                                                                                                                                                                                                            
 '    [42, "Dark grey background area with text"],\n'
 '    [43, "Text or icon on a dark grey background"],\n'                                                                     
 '    [44, "Dark grey background area"],\n'                                                                                  
 '    [45, "Dark grey background area with text"],\n'                                                                        
 '    [46, "Dark grey background area with text"],\n'                                                                        
 '    [47, "Button with text or icon"],\n'                                                                                   
 '    [48, "Button with \'+\'"],\n'                                                                                          
 '    [49, "Dark grey background area with text"],\n'                                                                        
 '    [50, "Dark grey background area with text"],\n'                                                                        
 '    [51, "Dark grey background area with text"],\n'                                                                        
 '    [52, "Dark grey background area text"],\n'                                                                                                                                                                                                          
 '    [53, "Button with \'7\'"],\n'                                                                                          
 '    [54, "Dark grey background area"],\n'                                                                                  
 '    [55, "Dark grey background area with text or icon"],\n'                                                                
 '    [56, "Dark grey background area with icons"],\n'                                                                       
 '    [57, "Orange button with \'÷\'"],\n'                                                                                   
 '    [58, "Text or icon"],\n'                                                                                               
 '    [59, "Dark grey background area with text or icon"],\n'                                                                
 '    [60, "Dark grey background area with text"],\n'                                                                        
 '    [61, "Dark grey background area with icon or text"],\n'                                                                
 '    [62, "Dark grey background area"],\n'                                                                                  
 '    [63, "Dark grey background area with text"],\n'                                                                        
 '    [64, "Column of buttons with icons"],\n'                                                                               
 '    [65, "Dark grey background area with icons"],\n'                                                                       
 '    [66, "Text or icon on a darker grey background"],\n'                                                                   
 '    [67, "Dark grey area with text or icon"],\n'                                                                           
 '    [68, "Dark grey background area"],\n'      

  File "/Users/abrichr/oa/OpenAdapt/openadapt/utils.py", line 601, in parse_code_snippet
    rval = ast.literal_eval(json_string)
           |   |            -> '{\n  "descriptions": [\n    [1, "A plus sign icon on an orange background"],\n    [2, "A blank gray square"],\n    [3, "The ...
           |   -> <function literal_eval at 0x104ba2ef0>
           -> <module 'ast' from '/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py'>

  File "/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py", line 64, in literal_eval
    node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
                     |     |              -> <method 'lstrip' of 'str' objects>
                     |     -> '{\n  "descriptions": [\n    [1, "A plus sign icon on an orange background"],\n    [2, "A blank gray square"],\n    [3, "The ...
                     -> <function parse at 0x104ba2cb0>

  File "/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
                   |       |         |     -> 1024
                   |       |         -> 'eval'
                   |       -> '<unknown>'
                   -> '{\n  "descriptions": [\n    [1, "A plus sign icon on an orange background"],\n    [2, "A blank gray square"],\n    [3, "The ...

  File "<unknown>", line 19
    }
    ^

SyntaxError: closing parenthesis '}' does not match opening parenthesis '[' on line 2

To Reproduce

 python -m openadapt.replay SegmentReplayStrategy "replay verbatim"

[abrichr]

https://chatgpt.com/share/e92a6524-bb9a-4692-ac2e-9e10fe52aeff

For now we will focus on reducing the number of segments per prompt in upstream logic.

[abrichr]

A regression was introduced in https://github.com/OpenAdaptAI/OpenAdapt/pull/679 in openadapt/adapters/ultralytics.py.

Before:

    conf: float = 0.4,
    iou: float = 0.9,

After:

    # threshold below which boxes will be filtered out
    conf: float = 0,
    # discards all overlapping boxes with IoU > iou_threshold
    iou: float = 0.05,

[abrichr]

The regression has been fixed in https://github.com/OpenAdaptAI/OpenAdapt/pull/758 but the issue will crop up again for lower NMS threshold values.