DanielHarrod / AppTestStudio

Visually, quickly, automate your apps, games, and android emulators.
GNU General Public License v3.0
26 stars 7 forks source link

Help understanding image detection #46

Open thorrenat opened 2 years ago

thorrenat commented 2 years ago

Hello Daniel,

First of all I want to thank you enourmously for this useful tool you designed, I am using it daily basis on my favourite game and after some learning I managed to automate it perfectly. Thanks.

In order to improve my scripts I've been trying to use image detection fuctions to enlarge my capabilities, sadly, I failed every time I tried.

I don't understand what I am doing wrong, I make objects, select mask, colour and at the time of running detections does nothing. It just walks through next action.

Would be possible for you to make a video or some extra documentation of how to use it?

Studio

Looking forward on hearing from you. Best regards R.

DanielHarrod commented 2 years ago

Hi R,

That's great you have been able to figure it out. There's quite a bit of cool functionality in ATS. Including script design.

I have a video in the works to cover many aspects of the functionality and script design. But it's a few weeks out.

Here's a couple hints on object search that should get you running.

Threshold is the relationship between the source and target image, each threshold will be different, because it's relative to the images being compared. Since it works at the individual RBG channel level, it's almost impossible to figure it out without using the Test button and tweaking it so that it meets the threshold at an acceptable level.

The object search compares the two images, first it takes a screenshot, then it crops the image (this is the area to search). Then it takes the object and does math at every location on the image pixel by pixel and compares each possible location. It takes the highest score it finds after comparing every possible location, then checks to see if it meets the Threshold. If it meets the threshold then....

If you have a click event directly under the object search, it will click on the located image - centering the click event box size on that location then click there.

Here's one On this event, the gold chest floats inside this range it goes up and down and scrolls left. So since it's only in that range, then I don't need to check anywhere else. image

Here's my action, The size of the click area is how wide the clicks will be. Also since it's moving I have a -9 pixel offset. image

It doesn't really matter where the box is, this is just configuring the click area and offset. This one and the above one work the same. image

AHH...but..What if you don't want to click on the object. Instead you want to click somewhere else, here's the hack for that. (This is an option I need to build into the properties, but I haven't had any idea how to make it intuitive(ish)).

Here's how you do it (if you don't want to click on the object you searched for). Object Search Event -> Event (empty or color/point) -> Click Action

VS.

Object Search Event -> Click Action

**** The most important part. See the Object doesn't have the background included, that's because the background changes on every level and when the level scrolls. And I don't want the background in the calculation. image

So this is: image

Is not as good as this: image

Object search takes considerably more CPU time to do the search than using color/points, so if you can do it with color/point that would be ideal or set the mask appropriately. (Also Mask isn't required)

Here's another example, This one I want to click "Buy" if it's on the screen, and a few other icons, and scroll down the list, then scroll back to the top if I don't take any actions. First check that it's viable before doing object searches. That way it won't run every time. image

Then If I see Buy, Click it. In the Village Group as it goes down the list, if it gets to the bottom, it performs scrolling actions if there's nothing found. image

Let me know if this helps,

Daniel

On Tue, Oct 25, 2022 at 5:28 AM thorrenat @.***> wrote:

Hello Daniel,

First of all I want to thank you enourmously for this useful tool you designed, I am using it daily basis on my favourite game and after some learning I managed to automate it perfectly. Thanks.

In order to improve my scripts I've been trying to use image detection fuctions to enlarge my capabilities, sadly, I failed every time I tried.

I don't understand what I am doing wrong, I make objects, select mask, colour and at the time of running detections does nothing. It just walks through next action.

Would be possible for you to make a video or some extra documentation of how to use it?

[image: Studio] https://user-images.githubusercontent.com/28702741/197749974-087c0e85-50ec-43c6-884c-ea59d762f22b.jpg

Looking forward on hearing from you. Best regards R.

— Reply to this email directly, view it on GitHub https://github.com/DanielHarrod/AppTestStudio/issues/46, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7QNOJZ77WYLS3WUWUTC7DWE6Y5HANCNFSM6AAAAAARN2L3S4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

thorrenat commented 2 years ago

Hi Daniel,

Thanks fot this complete explanation, unfortunately example pictures were not loaded. I will try all your proposals as soon as possible, and give you feedback.

Keep in touch

DanielHarrod commented 2 years ago

Hi R,

That's great you have been able to figure it out. There's quite a bit of cool functionality in ATS. Including script design.

I have a video in the works to cover many aspects of the functionality and script design. But it's a few weeks out.

Here's a couple hints on object search that should get you running.

Threshold is the relationship between the source and target image, each threshold will be different, because it's relative to the images being compared. Since it works at the individual RBG channel level, it's almost impossible to figure it out without using the Test button and tweaking it so that it meets the threshold at an acceptable level.

The object search compares the two images, first it takes a screenshot, then it crops the image (this is the area to search). Then it takes the object and does math at every location on the image pixel by pixel and compares each possible location. It takes the highest score it finds after comparing every possible location, then checks to see if it meets the Threshold. If it meets the threshold then....

If you have a click event directly under the object search, it will click on the located image - centering the click event box size on that location then click there.

Here's one On this event, the gold chest floats inside this range it goes up and down and scrolls left. So since it's only in that range, then I don't need to check anywhere else. image

Here's my action, The size of the click area is how wide the clicks will be. Also since it's moving I have a -9 pixel offset. image

It doesn't really matter where the box is, this is just configuring the click area and offset. This one and the above one work the same. image

AHH...but..What if you don't want to click on the object. Instead you want to click somewhere else, here's the hack for that. (This is an option I need to build into the properties, but I haven't had any idea how to make it intuitive(ish)).

Here's how you do it (if you don't want to click on the object you searched for). Object Search Event -> Event (empty or color/point) -> Click Action

VS.

Object Search Event -> Click Action

**** The most important part. See the Object doesn't have the background included, that's because the background changes on every level and when the level scrolls. And I don't want the background in the calculation. image

So this is: image

Is not as good as this: image

Object search takes considerably more CPU time to do the search than using color/points, so if you can do it with color/point that would be ideal or set the mask appropriately. (Also Mask isn't required)

Here's another example, This one I want to click "Buy" if it's on the screen, and a few other icons, and scroll down the list, then scroll back to the top if I don't take any actions. First check that it's viable before doing object searches. That way it won't run every time. image

Then If I see Buy, Click it. In the Village Group as it goes down the list, if it gets to the bottom, it performs scrolling actions if there's nothing found. image

Let me know if this helps,

Daniel

thorrenat commented 2 years ago

Hello Daniel,

I tried some of the described steps, I am getting an error message "The type initializer for 'OpenCvSharp.Mat' threw an exception" every time I test or run .

Any idea how to solve that?

Thanks

DanielHarrod commented 1 year ago

Hi thorrenat,

This is a tricky one, as I wasn't able to reproduce it on any workstation that I have.

The application is having difficulty loading the DLL's in the install directory under /dll/x64 and /dll/x86

You might try running ATS as an administrator image

...or making sure that the logged in user has security on those folders. image

The other alternative is to run ATS from source code (eg install visual studio, run it from source code)

thorrenat commented 1 year ago

Hi Daniel,

That may explain something, I don't have such a folder: image

The only folder is C:/Program Files/App Test Stidio/App Test Studio/bin

But no dll folder (not even in hiding folders)

Installed version 1.0.0.14 from 16 Jun 2022 image

DanielHarrod commented 1 year ago

Sorry, I didn't mean to say dll folder, I meant the bin folder. I think it's not finding the dll's here. You can copy them to the folder where AppTestStudio.exe is located and it will find them. image

After the copy. image

Let me know if this works.

thorrenat commented 1 year ago

This definetely made it work!!! Thanks for your time and dedication helping me here.

I want to invite you to a 6 pack or something, how can I compensate you for all this effort?

DanielHarrod commented 1 year ago

That's awesome.

Thanks for sticking with it. I'll try and see if this can be prevented in the future.

Daniel

On Mon, Nov 14, 2022 at 10:15 AM thorrenat @.***> wrote:

This definetely made it work!!! Thanks for your time and dedication helping me here.

I want to invite you to a 6 pack or something, how can I compensate you for all this effort?

— Reply to this email directly, view it on GitHub https://github.com/DanielHarrod/AppTestStudio/issues/46#issuecomment-1314009697, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7QNOJU4OV7H6JFEZTOSB3WIJQR5ANCNFSM6AAAAAARN2L3S4 . You are receiving this because you commented.Message ID: @.***>

thorrenat commented 1 year ago

No, seriously, your patreon page is not working, how can I compensate you?

DanielHarrod commented 1 year ago

That's fixed too, I appreciate your support.