search

appium / ruby_lib

:gem: Ruby library for Appium
http://rubygems.org/gems/appium_lib
216 stars 160 forks source link

Finding by images is not working as expected #851

Closed fruwe closed 5 years ago

fruwe commented 5 years ago

This is a

Summary

Environment

Tried with UIAutomator and UIAutomator2

Actual behaviour and steps to reproduce

I have an Android view, with nearly white background and one button containing an enlarged version of a 32x32px icon image. The button's bounds are [531,172][831,392]. I am trying to press the element containing that image.

What I tried is

base64_img = Base64.strict_encode64(File.open('m.png', 'r+b').read)

e = @driver.find_element(image: base64_icon_img)
#<Selenium::WebDriver::Element:0x4eebb91c id="appium-image-element-dca0cf30-0195-484a-ac1c-55adb042e4a9">

e.rect
#<struct Selenium::WebDriver::Rectangle x=183, y=312, width=32, height=32>

The problem is that in this area is no element, I tried to click it but it is not working, nil is returned. (In case of UIAutomator2 "OK" is returned)

Furthermore, I tried to get an image of the element or other information, so I tried find_element_by_image.

(byebug) appium_driver.find_element_by_image('m.png')
*** Selenium::WebDriver::Error::UnknownError Exception: An unknown server-side error occurred while processing the command. Original error: Cannot find any occurences of the partial image in the full image. Original error: OpenCV Error: ((depth == CV_8U || depth == CV_32F) && type == _templ.type() && _img.dims() <= 2) in cv::matchTemplate, in file C:\Users\xx\AppData\Roaming\npm\node_modules\opencv4nodejs\node_modules\opencv-build\opencv\opencv\modules\imgproc\src\templmatch.cpp, line 1102, status -215

nil

I also tried (as explained in the documentation) to output an image like this:

(byebug) appium_driver.find_element_by_image('m.png', visualize: true)
*** ArgumentError Exception: wrong number of arguments (given 2, expected 1)

This causes an error, since the core library only declares a method with a single parameter.

Expected behaviour

e = @driver.find_element(image: base64_icon_img)

should return nil/error if not found or the element containing the image. It should never return an element not mentioned in the page_source (or maybe in the page_source's of all contexts).

Link to Appium/Ruby logs

Any additional comments

find_element_by_image contained originally more parameters, but they were removed here: https://github.com/appium/ruby_lib_core/commit/590e7981f924524ba40b9d918b41af22cf14cc59

fruwe commented 5 years ago

However, I could find and click the element using a cropped screenshot.

(byebug)  e = appium_driver.find_element(image: cropped_icon_img)
#<Selenium::WebDriver::Element:0x..fcbc7ceedb89e4650 id="appium-image-element-1cca953b-2b38-485b-a372-27de01f0ad98">
(byebug) e.rect
#<struct Selenium::WebDriver::Rectangle x=534, y=150, width=310, height=260>
(byebug) e.click
nil
KazuCocoa commented 5 years ago

from appium/ruby_lib_core@590e798 , the finding image logic has moved to https://github.com/appium/appium/blob/master/docs/en/advanced-concepts/image-elements.md

The previous implementation was the experimental one in client-side. The new one was handled the same logic in appium-base-driver. So, for now, the process depends on Appium base driver's logic.

You can check what image is returned by the appium server using https://github.com/appium/ruby_lib_core/blob/ffe14bd8263becfc4cac1446dba62ca6544c886b/lib/appium_lib_core/common/device/image_comparison.rb#L85 . Internally, the previous implementation used the image comparison API which already implemented in ruby lib

KazuCocoa commented 5 years ago

https://www.rubydoc.info/github/appium/ruby_lib_core/Appium/Core/Base/Device/ImageComparison

KazuCocoa commented 5 years ago

ah, sorry. I misunderstood. I should have updated ruby_lib side.

KazuCocoa commented 5 years ago

✅ v10.2.0

thanks for the report!