search

block / goose

Goose is a developer agent that operates from your command line to help you do the boring stuff.
https://block.github.io/goose/
Apache License 2.0
109 stars 17 forks source link

fix: Resize file in screen toolkit #81

Closed zakiali closed 1 month ago

zakiali commented 1 month ago

Updating the screen toolkit to have the capability to resize the image.

This is a solution for errors in payload size when sending images, for example. We've been seeing this in some providers, but not all. The limits in OpenaAI for payload size is 20MB.

By tools default functionality doesn't change, only when prompted to resize the image if you run into errors.

edit: Additionally, wanted to note that the file size isn't the right number for the payload since the images are being base64 encoded, inflating the disk size by 37%.

michaelneale commented 1 month ago

@zakiali any thought on a ride for this PR: make it screenshot all screens?

zakiali commented 1 month ago

@zakiali any thought on a ride for this PR: make it screenshot all screens?

Added options to take screenshots of diff screens instead of the main and it's a pretty easy extension to do all screens (pass in more file paths to the main command). I left it out though because it might be too much and maybe an extended toolkit that does that for power use cases?

michaelneale commented 1 month ago

hrm - was able to test it, but when asking it to take it of both screens and mention size, saw this error:

image

michaelneale commented 1 month ago

the parameter thing may be different, but I also saw this (thought I could try 2 screens): 

G❯ can you take a look at both my screens and tell me what you see

screencapture -x -D 1 /tmp/goose_screenshot_4cc6811760b04cc7984317fef33c02b2.png  

screencapture -x -D 2 /tmp/goose_screenshot_aa42b9aabaf94fbb814259f669c2a5c8.png 
Traceback (most recent call last):
  File "/Users/micn/Documents/code/domestic-goose/.venv/lib/python3.12/site-packages/exchange/providers/utils.py", line 30, in raise_for_status
    response.raise_for_status()
  File "/Users/micn/Documents/code/domestic-goose/.venv/lib/python3.12/site-packages/httpx/_models.py", line 763, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.openai.com/v1/chat/completions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

it may be stuffing 2 images at once to openai - if you like I can take a look and try to push some fixes to this branch?