Closed zakiali closed 1 month ago
@zakiali any thought on a ride for this PR: make it screenshot all screens?
@zakiali any thought on a ride for this PR: make it screenshot all screens?
Added options to take screenshots of diff screens instead of the main and it's a pretty easy extension to do all screens (pass in more file paths to the main command). I left it out though because it might be too much and maybe an extended toolkit that does that for power use cases?
hrm - was able to test it, but when asking it to take it of both screens and mention size, saw this error:
the parameter thing may be different, but I also saw this (thought I could try 2 screens):
G❯ can you take a look at both my screens and tell me what you see
screencapture -x -D 1 /tmp/goose_screenshot_4cc6811760b04cc7984317fef33c02b2.png
screencapture -x -D 2 /tmp/goose_screenshot_aa42b9aabaf94fbb814259f669c2a5c8.png
Traceback (most recent call last):
File "/Users/micn/Documents/code/domestic-goose/.venv/lib/python3.12/site-packages/exchange/providers/utils.py", line 30, in raise_for_status
response.raise_for_status()
File "/Users/micn/Documents/code/domestic-goose/.venv/lib/python3.12/site-packages/httpx/_models.py", line 763, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.openai.com/v1/chat/completions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
it may be stuffing 2 images at once to openai - if you like I can take a look and try to push some fixes to this branch?
Updating the screen toolkit to have the capability to resize the image.
This is a solution for errors in payload size when sending images, for example. We've been seeing this in some providers, but not all. The limits in OpenaAI for payload size is 20MB.
By tools default functionality doesn't change, only when prompted to resize the image if you run into errors.
edit: Additionally, wanted to note that the file size isn't the right number for the payload since the images are being base64 encoded, inflating the disk size by 37%.