Is there a better way to distinguish between pale and fat drawn pictures?

[LisBerndt]

Hello!

We can only distinguish between the these two pictures, when we set Settings.MinSimilarity =1. But this causes that sometimes other pictures however are not recognized. Is there another option to distinguish successfully?

[balmma]

You can do a find() or a wait() and check the score on the returned Match.

e.g. (pseudo code)

while True:
  match = wait(pattern.png)
  if match.score >= 0.95:
    print "match"
    break

You most probably need an additional measure (e.g. timeout) to terminate the infinite loop after some tries.

[balmma]

Or if you want to perform a click:

while True:
  match = wait(pattern.png)
  if match.score >= 0.95:
    match.click()
    break

[balmma]

But why don't you set the similarity directly on the pattern?

[balmma]

something like:

wait(Pattern("1575991793670.png").similar(1.0))

[ghost]

You got me wrong. This is one button which can have one of these states. Only with the fat state a click shall happen. By design our framework wrapping Sikulix does not allow to set the minSim. for a specific image, rather it is set once. Only with minSim. =1 it works as expected. Unfortunately, in turn it may happen that other images are not recognized that should be recognized. Probably I'm loooking for non-existing middle way?

[balmma]

Ok, seems to be a bit inflexible to me. Don't you use the regular SikuliX functions?

[balmma]

BTW: Your images appear to be approximately 99.5 % similar to SikuliX. The score of the match is 0.995. That's why you have to set minSimilarity to 1.0. If you are not able to set it to the individual pattern, is it not possible to do a Settings.minSimilarity multiple times during your script run either?

[balmma]

BTW2: Setting the similarity individually per pattern is a core feature of SikuliX. Would probably be worth ro talk to the maintainer of the wrapper to support it as well.

[ghost]

I do. The framework command wraps the wait function combined with click function in the same way you described above. Implementing a new command or parameter just to make Sikulix for special cases clear that it shall search with the highest minSim, would be an option that would realize only reluctantly; the number of commands and parameters shall be as small as possible.

[ghost]

If you are not able to set it to the individual pattern, is it not possible to do a Settings.minSimilarity multiple times during your script run either?

Got the same idea. I'll try it out.

[balmma]

Implementing a new command or parameter just to make Sikulix for special cases clear that it shall search with the highest minSim

Not with the highest minSim but rather with a arbitrary similarity. In our scripts we usually use various fine tuned similarities for different patterns within the same script. The framework would be much more flexible allowing this as well.

Just out of curiosity: Is the wrapper framework you are using a 3rd party product or something you write on your own?

[LisBerndt]

Not with the highest minSim but rather with a arbitrary similarity. In our scripts we usually use various fine tuned similarities for different patterns within the same script. The framework would be much more flexible allowing this as well.

My similar idea is, to set the minSim depending on what action is executed; that is, for click-action minSim must have the highest value. For result recognition a smaller value should be sufficient.

[balmma]

IMHO the optimal similarity is not really depending on the action but rather on the specific pattern and other similar stuff on the screen.

[LisBerndt]

Just out of curiosity: Is the wrapper framework you are using a 3rd party product or something you write on your own?

It´s a self programmed ruby-script which wraps in different apps to do some actions; e.g. communicate with an eval-board, rest-interface. Selenium is wrapped in also to test a external web-app. All actions to do are definded in a csv-file. Due to its grown structure it´s somewhat inflexible, but very effective to test our various products on our embedded system.

[balmma]

Due to its grown structure it´s somewhat inflexible

I see, then probably it would really be the simplest to do a Settings.minSimilarity before the action and revert it afterwards. I don't believe that you are going to be happy with your action based approach.

[LisBerndt]

IMHO the optimal similarity is not really depending on the action but rather on the specific pattern and other similar stuff on the screen.

Yet the action executed is depending what pattern is adressed. Most patterns are only checked for appearance after executing some actions, for them a smaller value is sufficient. Patterns like the above ones are generally supposed to be clicked - or not. As seen we ne need the highest value for them.

[balmma]

As seen we ne need the highest value for them.

You only need the highest value if the image is similar to another image you don't want to click. Otherwise a smaller similarity is fine as well. It might very well be that you want to wait for a specific image and have another very similar image on the screen (and not click on it).

But hey, if the action based approach is working for your use case just do it.

[balmma]

I guess the issue can be closed. It seems to be rather a limitation of the wrapper framework than a limitation of SikuliX.

@RaiMan We probably have to check why such images are THAT similar to SikuliX. AFAICS in such a case (plain color, no mask) we use TM_CCOEFF_NORMED method for the matchTemplate call. This method seems to be not very accurate when it comes to slight variations in pixel intensity. This might even be the reason that we can't really find fields with a specific color with SikuliX. But I currently have no clue what might give better results.

[LisBerndt]

I just recognized, that I didn´t inspect the case how Sikulix behaves when it´s searching for the fat pattern at minSim =1.0 --> It never finds it. Even with a fresh recorded png.-pattern or when I delimit the search-region to the pattern position/dimensions, Sikulix fails. MinSim at 0.99 works.

[balmma]

MinSim at 0.99 works.

But then it also finds the grey one, doesn't it?

[LisBerndt]

MinSim at 0.99 works.

But then it also finds the grey one, doesn't it?

Yes.

[balmma]

Finding the grey image using the black pattern gives me a score of 0.997876524925. Means that you might be fine setting minSimilarity to 0.999?

[LisBerndt]

Already tried. --> fat is clicked, but also pale.

[LisBerndt]

But hey, 0.9999 works!

[balmma]

0.9999 works

Seems to be a bit shaky. But alright if it works for u :-)