wrong coordinate

[daaniyaan]

I asked it to play Spotify and it guessed the play bottom at x 78% y 46% which is wrong.

maybe for a more detailed guess we can have more gridlines? something like this maybe

[daaniyaan]

tried setting grid_interval as 100 instead of 500. still no success. it guess as 50 93 which is again not correct.

[Mekcyed]

@rohanarun Can you please stop spamming your message under EVERY Issue. Thanks

[Mekcyed]

@daaniyaan I tested today another apporach. I hink it would be best to use a multi-step-apporach.

1. Use a wider Grid to let GPT Vision define Areas of Intereset and return wider coordinates.
2. Use these coordinates to crop the screenshot and let GPT Vision do a better guess of find the needed buttons etc.

With a wide Image as the fullscreen Images it might be necessary to do this apporach not only once?

[daaniyaan]

@daaniyaan I tested today another apporach. I hink it would be best to use a multi-step-apporach.

1. Use a wider Grid to let GPT Vision define Areas of Intereset and return wider coordinates.
2. Use these coordinates to crop the screenshot and let GPT Vision do a better guess of find the needed buttons etc.

With a wide Image as the fullscreen Images it might be necessary to do this apporach not only once?

this could be a good approach. how many times did you test it? noticed improvement?

[Mekcyed]

Tested quite a lot today. I noticed that it can handle wider grid sizes better.

Had this as first image:

And this after cropping it to the area of interest:

[Mekcyed]

Dude I am just a random guy who finds the idea of using got vison interesting. Didn't even contribute a single thing to this repo.

Chill down

[joshbickett]

wide Image as the fullscreen Images it might be necessary to do this apporach not only once?

@Mekcyed This is indeed an interesting approach. I recommend checking out this PR that is working on something similar: https://github.com/OthersideAI/self-operating-computer/pull/57

If you want to work on this initiative, I'd love to see a PR. Essentially you could do a "binary-search" on the screen, cutting it into ever smaller pieces to find the right coordinance to click!

[joshbickett]

Sorry about @rohanarun, I've blocked him from this repo now

[joshbickett]

@Mekcyed @daaniyaan we created a discord channel for Self-operating computer on our Discord here if you want to join and discuss more. Great seeing all these ideas!

[joshbickett]

Let me know if you guys are done with this discuss here and want to move to discord and I'll close this ticket

[joshbickett]

I've not head more on this discussion so I'll close this ticket for now. Feel free to open another issue if you have one or join our discord to continue the discussion.