[brainstorming] - how to detect WHY Screen recognition fails?

[andreabisello]

This is my use case

with seleniumlibrary, i take screenshot of the web browser.

engineering team release a new version

with sikuli i run test in order to undertand if something is different.

some area should to be different and this is ok,

some other area doesn't have to change, so if there are some change there are some regression.

good.

some time i'm in difficult to understand WHY a test fail versus image recognition.

if the image comparison was against the same screen area and same resolution , i can use pillow to make a image diff, but with sikuli, that search for an image everywhere in the screen, how i can obtain this result? any suggestion?

[rainmanwy]

@abioneperhobby sikuli could be set to find specific area. sikuli java doc User could define a region, and "region" support search and other operations.

sorry, I am not quite understand your requirements. If want to compare one image(captured whole screen) with screen, and find the difference, maybe could split image and screen to several region, and compare each regions to get matched score.

And this let me to think, sikulilibrary could set "minSimilarity" with low score, and use "findAll" to get all matched regions, and select the matched region with highest score. maybe this is more traceable to user, but with lower performance.

[andreabisello]

@rainmanwy create region can be useful to reduce screenshot comparison complexity. splitting area in zones can be useful to better understand where there is a different.

my requirement is to create A-B testing creating screenshot.

Robot framework-sikulilibrary can already use regions?

[rainmanwy]

@abioneperhobby , not yet. By default, this lib will search whole screen. But i think it is easy to implement keywords to define searched region. For example: setSearchedRegion(int x, int y, int width, int height) ""user define region"""" setSearchScreen() """set back to search whole region""" I am not sure where these keywords are enough and useful. How about your suggestion?

[andreabisello]

i think these keywords can be useful, also with Capture Region Screen(x,y,with,height)

the top should be permitting to obtain the region of the application window (for example browser or software window) , it will make recognition tasks easier