I tried to run the same tests on iOS Simulators which was successful on Android Emulators. But its unable to identify elements & below are the error details. Can you please guide & help resolve the issue here?
For Element Lookup Mode:
`
Code used ::
$("findElementByCustom :::----> Search icon using AI");
getAppDriver().findElement(MobileBy.custom("ai:search")).click();
Capabilities set
capabilities.setCapability("customFindModules", customFindModules);
capabilities.setCapability("testaiFindMode", "element_lookup");
capabilities.setCapability("testaiObjectDetectionThreshold", "0.1");
capabilities.setCapability("shouldUseCompactResponses", false);
2020-01-29 13:13:04,144 120104 [TestNG-tests-1] INFO com.walmart.asda.mobile.poc.TestSearch_01 - Executing : findElementByCustom :::----> Search icon using AI
[HTTP] --> POST /wd/hub/session/787a5a29-2972-4377-b5c1-d354b07c8b6b/element
[HTTP] {"using":"-custom","value":"ai:search"}
[debug] [W3C (787a5a29)] Calling AppiumDriver.findElement() with args: ["-custom","ai:search","787a5a29-2972-4377-b5c1-d354b07c8b6b"]
[debug] [XCUITest] Executing command 'findElement'
[debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier'
2020-01-29 13:13:06.149943: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-29 13:13:06.183955: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x102b8ec80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-29 13:13:06.183991: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
[debug] [BaseDriver] Waiting up to 0 ms for condition
[ai] Retrieving current settings to check element response attributes
[ai] We will need to update settings to include element response attributes
[debug] [WD Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [WD Proxy] Proxying [POST /appium/settings] to [POST http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/appium/settings] with body: {"settings":{"elementResponseAttributes":"rect"}}
[debug] [WD Proxy] Got response with status 200: {"value":{"screenshotQuality":1,"mjpegScalingFactor":100,"elementResponseAttributes":"rect","shouldUseCompactResponses":false,"mjpegServerFramerate":10,"mjpegServerScreenshotQuality":25},"sessionId":"C8C31F6A-57DD-4036-9457-297962A87A38","status":0}
[ai] Setting confidence threshold to default value of 0.1
[ai] Retrieving data for all leaf-node elements on screen
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id
[debug] [BaseDriver] Waiting up to 0 ms for condition
[debug] [WD Proxy] Matched '/elements' to command name 'findElements'
[debug] [WD Proxy] Proxying [POST /elements] to [POST http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/elements] with body: {"using":"xpath","value":"//[not(child::)]"}
[debug] [WD Proxy] Got response with status 200: {"value":[{"ELEMENT":"2F000000-0000-0000-8852-010000000000","rect":{"y":0,"x":0,"width":375,"height":40}},{"ELEMENT":"3C000000-0000-0000-8852-010000000000","rect":{"y":50,"x":150,"width":75,"height":32}},{"ELEMENT":"3D000000-0000-0000-8852-010000000000","rect":{"y":44,"x":286,"width":73,"height":44}},{"ELEMENT":"A4000000-0000-0000-8852-010000000000","rect":{"y":88,"x":0,"width":375,"height":62}},{"ELEMENT":"A5000000-0000-0000-8852-010000000000","rect":{"y":88,"x":16,"width":359,"height":62}},{"ELEMENT":"C1000000-0000-0000-8852-010000000000","rect":{"y":170,"x":16,"width":119,"height":27}},{"ELEMENT":"C2000000-0000-0000-8852-010000000000","rect":{"y":196,"x":16,"width":266,"height":49}},{"ELEMENT":"C3000000-0000-0000-8852-010000000000","rect":{"y":248,"x":16,"width":343,"height":3}},{"ELEMENT":"C4000000-0000-0000-8852-010000000000","rect":{"y":254,"x":16,"width":214,"height":25}},{"ELEMENT":"D0000000-0000-0000-8852-010000000000","rect":{"y":308,"x":16,"width":164,"height":40}},{"ELEMENT":"D1000000-0000-0000...
[ai] Getting window size in case we need to scale screenshot
[debug] [WD Proxy] Proxying [GET /window/size] to [GET http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/window/size] with no body
[debug] [WD Proxy] Got response with status 200: "{\n \"value\" : {\n \"width\" : 375,\n \"height\" : 812\n },\n \"sessionId\" : \"C8C31F6A-57DD-4036-9457-297962A87A38\",\n \"status\" : 0\n}"
[ai] Getting screenshot to use for classifier
[debug] [XCUITest] Taking screenshot with WDA
[debug] [WD Proxy] Matched '/screenshot' to command name 'getScreenshot'
[debug] [WD Proxy] Proxying [GET /screenshot] to [GET http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/screenshot] with no body
[debug] [WD Proxy] Got response with status 200: "{\n \"value\" : \"iVBORw0KGgoAAAANSUhEUgAABGUAAAmECAIAAAAjR0TuAAAAAXNSR0IArs4c6QAA\r\nABxpRE9UAAAAAgAAAAAAAATCAAAAKAAABMIAAATCAAavDLBpvG4AAEAASURBVHgB\r\n7L0HfB7Fue9\/z\/3fc0\/LOScJ1V2W5YYxJUDoENKBAKElhBQIkBBOICS4qEuWuwED\r\nSSAhpN4AORwCBDf1atNC770GAqa6SZZkSd7\/95nnfcdrWRjZ4AL+7We9np2dnZn9\r\n7uyr57fPzOz\/+l9aREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREA...
[ai] Turning screenshot into HTML image for use with canvas
[ai] Screenshot and screen size did not match. Screen size is 375x812 but screenshot size is 1125x2436. Scaled screenshot to match screen size
[ai] Getting screenshot slices for each element
[ai] Resetting element response attribute setting to original value: "type,label"
[debug] [WD Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [WD Proxy] Proxying [POST /appium/settings] to [POST http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/appium/settings] with body: {"settings":{"elementResponseAttributes":"type,label"}}
[debug] [WD Proxy] Got response with status 200: {"value":{"screenshotQuality":1,"mjpegScalingFactor":100,"elementResponseAttributes":"type,label","shouldUseCompactResponses":false,"mjpegServerFramerate":10,"mjpegServerScreenshotQuality":25},"sessionId":"C8C31F6A-57DD-4036-9457-297962A87A38","status":0}
[debug] [W3C (787a5a29)] Encountered internal error running command: TypeError: Cannot destructure property x of 'undefined' or 'null'.
[debug] [W3C (787a5a29)] at getCanvasByRect (/usr/local/lib/node_modules/test-ai-classifier/lib/image.js:57:9)
[debug] [W3C (787a5a29)] at map (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:171:28)
[debug] [W3C (787a5a29)] at Array.map ()
[debug] [W3C (787a5a29)] at asyncmap (/usr/local/lib/node_modules/test-ai-classifier/node_modules/asyncbox/lib/asyncbox.js:74:26)
[debug] [W3C (787a5a29)] at getElementImages (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:169:30)
[debug] [W3C (787a5a29)] at findViaElementScreenshots (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:303:32)
[debug] [W3C (787a5a29)] at Object.find (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:377:12)
[debug] [W3C (787a5a29)] at condition (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/find.js:159:16)
[debug] [W3C (787a5a29)] at wrappedCondFn (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/timeout.js:157:12)
[HTTP] <-- POST /wd/hub/session/787a5a29-2972-4377-b5c1-d354b07c8b6b/element 500 43510 ms - 611
[HTTP]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:239)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
at io.appium.java_client.DefaultGenericMobileDriver.execute(DefaultGenericMobileDriver.java:41)
at io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.execute(IOSDriver.java:1)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:322)
at io.appium.java_client.DefaultGenericMobileDriver.findElement(DefaultGenericMobileDriver.java:61)
at io.appium.java_client.AppiumDriver.findElement(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.findElement(IOSDriver.java:1)
at io.appium.java_client.FindsByCustom.findElementByCustom(FindsByCustom.java:38)
at io.appium.java_client.MobileBy$ByCustom.findElement(MobileBy.java:623)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:314)
at io.appium.java_client.DefaultGenericMobileDriver.findElement(DefaultGenericMobileDriver.java:57)
at io.appium.java_client.AppiumDriver.findElement(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.findElement(IOSDriver.java:1)
at com.walmart.asda.mobile.poc.TestSearch_01.testBasicSearch_PhoneAndClickFav(TestSearch_01.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:584)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:172)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:804)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:145)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
`
For Object Detection Mode:
`
Code used ::
$("findElementByCustom :::----> Search icon using AI");
getAppDriver().findElement(MobileBy.custom("ai:search")).click();
Capabilities set
capabilities.setCapability("customFindModules", customFindModules);
capabilities.setCapability("testaiFindMode", "element_lookup");
capabilities.setCapability("testaiObjectDetectionThreshold", "0.1");
capabilities.setCapability("shouldUseCompactResponses", false);
2020-01-29 13:42:08,349 120921 [TestNG-tests-1] INFO com.walmart.asda.mobile.poc.TestSearch_01 - Executing : findElementByCustom :::----> Search icon using AI
[HTTP] --> POST /wd/hub/session/8653e25f-6cd0-4603-b4c2-a942bf0c2f77/element
[HTTP] {"using":"-custom","value":"ai:search"}
[debug] [W3C (8653e25f)] Calling AppiumDriver.findElement() with args: ["-custom","ai:search","8653e25f-6cd0-4603-b4c2-a942bf0c2f77"]
[debug] [XCUITest] Executing command 'findElement'
[debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier'
2020-01-29 13:42:10.518184: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-29 13:42:10.544574: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x10c8c5770 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-29 13:42:10.544601: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
[debug] [BaseDriver] Waiting up to 0 ms for condition
[ai] Will use object detection method for finding elements via AI
[ai] Setting confidence threshold to default value of 0.1
[ai] Getting window size in case we need to scale screenshot
[debug] [WD Proxy] Proxying [GET /window/size] to [GET http://localhost:46146/session/27188DFA-B718-41AD-AEF5-B4FCBAFDDBB4/window/size] with no body
[debug] [WD Proxy] Got response with status 200: "{\n \"value\" : {\n \"width\" : 375,\n \"height\" : 812\n },\n \"sessionId\" : \"27188DFA-B718-41AD-AEF5-B4FCBAFDDBB4\",\n \"status\" : 0\n}"
[ai] Getting screenshot to use for classifier
[debug] [XCUITest] Taking screenshot with WDA
[debug] [WD Proxy] Matched '/screenshot' to command name 'getScreenshot'
[debug] [WD Proxy] Proxying [GET /screenshot] to [GET http://localhost:46146/session/27188DFA-B718-41AD-AEF5-B4FCBAFDDBB4/screenshot] with no body
[debug] [WD Proxy] Got response with status 200: "{\n \"value\" : \"iVBORw0KGgoAAAANSUhEUgAABGUAAAmECAIAAAAjR0TuAAAAAXNSR0IArs4c6QAA\r\nABxpRE9UAAAAAgAAAAAAAATCAAAAKAAABMIAAATCAAavbYrc7aAAAEAASURBVHgB\r\n7J0HnB5Vvfdf3\/fe973FexWlpSebTSOEIiBdRK4FFJAqoleqIioipGzfzaaRAAFU\r\nVNSLeilepQTStj9bEpr03qtECISQutnd7G7m\/f7P\/3lOnuxuQgrJRvKbz2Ry5syZ\r\nc\/7znTnP\/n9zyvyv\/6VFBERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERAB...
[ai] Turning screenshot into HTML image for use with canvas
[ai] Screenshot and screen size did not match. Screen size is 375x812 but screenshot size is 1125x2436. Scaled screenshot to match screen size
[ai] Writing screenshot to /usr/local/lib/node_modules/test-ai-classifier/build-js/lib/Wed, 29 Jan 2020 08:12:11 GMT_classifiertest.png for use in object detection
[ai] Gathering object candidates from tensorflow native
2020-01-29 13:42:11.456773: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /usr/local/lib/node_modules/test-ai-classifier/model/obj_detection_model
2020-01-29 13:42:11.756360: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-01-29 13:42:12.260225: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-01-29 13:42:12.261404: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: /usr/local/lib/node_modules/test-ai-classifier/model/obj_detection_model/variables/variables.index
2020-01-29 13:42:12.261778: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 805000 microseconds.
[ai] Candidates retrieved
[ai] Making label predictions based on element images
This model execution did not contain any nodes with control flow or dynamic output shapes. You can use model.execute() instead.
[ai] Predictions for 34 element(s) took 0.686 seconds
[ai] Found 1 matching elements
[ai] Highest confidence of any element for desired label 'search' was 0.9994727373123169
[ai] Registering image elements with driver
[debug] [W3C (8653e25f)] Encountered internal error running command: TypeError: driver.registerImageElement is not a function
[debug] [W3C (8653e25f)] at map (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:365:19)
[debug] [W3C (8653e25f)] at Array.map ()
[debug] [W3C (8653e25f)] at findByObjectDetector (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:357:24)
[debug] [W3C (8653e25f)] at Object.find (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:375:12)
[debug] [W3C (8653e25f)] at condition (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/find.js:159:16)
[debug] [W3C (8653e25f)] at wrappedCondFn (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/timeout.js:157:12)
[HTTP] <-- POST /wd/hub/session/8653e25f-6cd0-4603-b4c2-a942bf0c2f77/element 500 17780 ms - 587
[HTTP]
org.openqa.selenium.WebDriverException: An unknown server-side error occurred while processing the command. Original error: driver.registerImageElement is not a function
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: 'm-c02xf25rjg5h', ip: 'fe80:0:0:0:1c5b:4ba:7301:ce39%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '1.8.0_192'
Driver info: io.appium.java_client.ios.IOSDriver
Capabilities {app: /Users/kverma1/Documents/mo..., automationName: XCUITest, browserName: , customFindModules: {ai: test-ai-classifier}, databaseEnabled: false, deviceName: iPhone X, fullReset: true, javascriptEnabled: true, locationContextEnabled: false, networkConnectionEnabled: false, newCommandTimeout: 120, noReset: false, platform: MAC, platformName: iOS, platformVersion: 12.0, shouldUseCompactResponses: false, takesScreenshot: true, testaiFindMode: object_detection, testaiObjectDetectionThreshold: 0.4, udid: E18161C4-00ED-481A-9281-867..., useNewWDA: false, wdaLocalPort: 46146, webStorageEnabled: false}
Session ID: 8653e25f-6cd0-4603-b4c2-a942bf0c2f77
*** Element info: {Using=-custom, value=ai:search}
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:239)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
at io.appium.java_client.DefaultGenericMobileDriver.execute(DefaultGenericMobileDriver.java:41)
at io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.execute(IOSDriver.java:1)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:322)
at io.appium.java_client.DefaultGenericMobileDriver.findElement(DefaultGenericMobileDriver.java:61)
at io.appium.java_client.AppiumDriver.findElement(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.findElement(IOSDriver.java:1)
at io.appium.java_client.FindsByCustom.findElementByCustom(FindsByCustom.java:38)
at io.appium.java_client.MobileBy$ByCustom.findElement(MobileBy.java:623)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:314)
at io.appium.java_client.DefaultGenericMobileDriver.findElement(DefaultGenericMobileDriver.java:57)
at io.appium.java_client.AppiumDriver.findElement(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.findElement(IOSDriver.java:1)
at com.walmart.asda.mobile.poc.TestSearch_01.testBasicSearch_PhoneAndClickFav(TestSearch_01.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:584)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:172)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:804)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:145)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
bigmauri
commented
3 years ago
Hi @jlipps ,
I tried to run the same tests on iOS Simulators which was successful on Android Emulators. But its unable to identify elements & below are the error details. Can you please guide & help resolve the issue here?
For Element Lookup Mode:
` Code used :: $("findElementByCustom :::----> Search icon using AI"); getAppDriver().findElement(MobileBy.custom("ai:search")).click();
Capabilities set capabilities.setCapability("customFindModules", customFindModules); capabilities.setCapability("testaiFindMode", "element_lookup"); capabilities.setCapability("testaiObjectDetectionThreshold", "0.1"); capabilities.setCapability("shouldUseCompactResponses", false);
2020-01-29 13:13:04,144 120104 [TestNG-tests-1] INFO com.walmart.asda.mobile.poc.TestSearch_01 - Executing : findElementByCustom :::----> Search icon using AI [HTTP] --> POST /wd/hub/session/787a5a29-2972-4377-b5c1-d354b07c8b6b/element [HTTP] {"using":"-custom","value":"ai:search"} [debug] [W3C (787a5a29)] Calling AppiumDriver.findElement() with args: ["-custom","ai:search","787a5a29-2972-4377-b5c1-d354b07c8b6b"] [debug] [XCUITest] Executing command 'findElement' [debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier' 2020-01-29 13:13:06.149943: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-01-29 13:13:06.183955: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x102b8ec80 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-01-29 13:13:06.183991: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version [debug] [BaseDriver] Waiting up to 0 ms for condition [ai] Retrieving current settings to check element response attributes [ai] We will need to update settings to include element response attributes [debug] [WD Proxy] Matched '/appium/settings' to command name 'updateSettings' [debug] [WD Proxy] Proxying [POST /appium/settings] to [POST http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/appium/settings] with body: {"settings":{"elementResponseAttributes":"rect"}} [debug] [WD Proxy] Got response with status 200: {"value":{"screenshotQuality":1,"mjpegScalingFactor":100,"elementResponseAttributes":"rect","shouldUseCompactResponses":false,"mjpegServerFramerate":10,"mjpegServerScreenshotQuality":25},"sessionId":"C8C31F6A-57DD-4036-9457-297962A87A38","status":0} [ai] Setting confidence threshold to default value of 0.1 [ai] Retrieving data for all leaf-node elements on screen [debug] [BaseDriver] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id [debug] [BaseDriver] Waiting up to 0 ms for condition [debug] [WD Proxy] Matched '/elements' to command name 'findElements' [debug] [WD Proxy] Proxying [POST /elements] to [POST http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/elements] with body: {"using":"xpath","value":"//[not(child::)]"} [debug] [WD Proxy] Got response with status 200: {"value":[{"ELEMENT":"2F000000-0000-0000-8852-010000000000","rect":{"y":0,"x":0,"width":375,"height":40}},{"ELEMENT":"3C000000-0000-0000-8852-010000000000","rect":{"y":50,"x":150,"width":75,"height":32}},{"ELEMENT":"3D000000-0000-0000-8852-010000000000","rect":{"y":44,"x":286,"width":73,"height":44}},{"ELEMENT":"A4000000-0000-0000-8852-010000000000","rect":{"y":88,"x":0,"width":375,"height":62}},{"ELEMENT":"A5000000-0000-0000-8852-010000000000","rect":{"y":88,"x":16,"width":359,"height":62}},{"ELEMENT":"C1000000-0000-0000-8852-010000000000","rect":{"y":170,"x":16,"width":119,"height":27}},{"ELEMENT":"C2000000-0000-0000-8852-010000000000","rect":{"y":196,"x":16,"width":266,"height":49}},{"ELEMENT":"C3000000-0000-0000-8852-010000000000","rect":{"y":248,"x":16,"width":343,"height":3}},{"ELEMENT":"C4000000-0000-0000-8852-010000000000","rect":{"y":254,"x":16,"width":214,"height":25}},{"ELEMENT":"D0000000-0000-0000-8852-010000000000","rect":{"y":308,"x":16,"width":164,"height":40}},{"ELEMENT":"D1000000-0000-0000... [ai] Getting window size in case we need to scale screenshot [debug] [WD Proxy] Proxying [GET /window/size] to [GET http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/window/size] with no body [debug] [WD Proxy] Got response with status 200: "{\n \"value\" : {\n \"width\" : 375,\n \"height\" : 812\n },\n \"sessionId\" : \"C8C31F6A-57DD-4036-9457-297962A87A38\",\n \"status\" : 0\n}" [ai] Getting screenshot to use for classifier [debug] [XCUITest] Taking screenshot with WDA [debug] [WD Proxy] Matched '/screenshot' to command name 'getScreenshot' [debug] [WD Proxy] Proxying [GET /screenshot] to [GET http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/screenshot] with no body [debug] [WD Proxy] Got response with status 200: "{\n \"value\" : \"iVBORw0KGgoAAAANSUhEUgAABGUAAAmECAIAAAAjR0TuAAAAAXNSR0IArs4c6QAA\r\nABxpRE9UAAAAAgAAAAAAAATCAAAAKAAABMIAAATCAAavDLBpvG4AAEAASURBVHgB\r\n7L0HfB7Fue9\/z\/3fc0\/LOScJ1V2W5YYxJUDoENKBAKElhBQIkBBOICS4qEuWuwED\r\nSSAhpN4AORwCBDf1atNC770GAqa6SZZkSd7\/95nnfcdrWRjZ4AL+7We9np2dnZn9\r\n7uyr57fPzOz\/+l9aREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREA... [ai] Turning screenshot into HTML image for use with canvas [ai] Screenshot and screen size did not match. Screen size is 375x812 but screenshot size is 1125x2436. Scaled screenshot to match screen size [ai] Getting screenshot slices for each element [ai] Resetting element response attribute setting to original value: "type,label" [debug] [WD Proxy] Matched '/appium/settings' to command name 'updateSettings' [debug] [WD Proxy] Proxying [POST /appium/settings] to [POST http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/appium/settings] with body: {"settings":{"elementResponseAttributes":"type,label"}} [debug] [WD Proxy] Got response with status 200: {"value":{"screenshotQuality":1,"mjpegScalingFactor":100,"elementResponseAttributes":"type,label","shouldUseCompactResponses":false,"mjpegServerFramerate":10,"mjpegServerScreenshotQuality":25},"sessionId":"C8C31F6A-57DD-4036-9457-297962A87A38","status":0} [debug] [W3C (787a5a29)] Encountered internal error running command: TypeError: Cannot destructure property)
[debug] [W3C (787a5a29)] at asyncmap (/usr/local/lib/node_modules/test-ai-classifier/node_modules/asyncbox/lib/asyncbox.js:74:26)
[debug] [W3C (787a5a29)] at getElementImages (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:169:30)
[debug] [W3C (787a5a29)] at findViaElementScreenshots (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:303:32)
[debug] [W3C (787a5a29)] at Object.find (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:377:12)
[debug] [W3C (787a5a29)] at condition (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/find.js:159:16)
[debug] [W3C (787a5a29)] at wrappedCondFn (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/timeout.js:157:12)
[HTTP] <-- POST /wd/hub/session/787a5a29-2972-4377-b5c1-d354b07c8b6b/element 500 43510 ms - 611
[HTTP]
xof 'undefined' or 'null'. [debug] [W3C (787a5a29)] at getCanvasByRect (/usr/local/lib/node_modules/test-ai-classifier/lib/image.js:57:9) [debug] [W3C (787a5a29)] at map (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:171:28) [debug] [W3C (787a5a29)] at Array.map (org.openqa.selenium.WebDriverException: An unknown server-side error occurred while processing the command. Original error: Cannot destructure property
xof 'undefined' or 'null'. Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03' System info: host: 'm-c02xf25rjg5h', ip: 'fe80:0:0:0:1c5b:4ba:7301:ce39%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '1.8.0_192' Driver info: io.appium.java_client.ios.IOSDriver Capabilities {app: /Users/kverma1/Documents/mo..., automationName: XCUITest, browserName: , customFindModules: {ai: test-ai-classifier}, databaseEnabled: false, deviceName: iPhone X, fullReset: true, javascriptEnabled: true, locationContextEnabled: false, networkConnectionEnabled: false, newCommandTimeout: 120, noReset: false, platform: MAC, platformName: iOS, platformVersion: 12.0, shouldUseCompactResponses: false, takesScreenshot: true, testaiFindMode: element_lookup, testaiObjectDetectionThreshold: 0.1, udid: E18161C4-00ED-481A-9281-867..., useNewWDA: false, wdaLocalPort: 49049, webStorageEnabled: false} Session ID: 787a5a29-2972-4377-b5c1-d354b07c8b6b *** Element info: {Using=-custom, value=ai:search}`
For Object Detection Mode: ` Code used :: $("findElementByCustom :::----> Search icon using AI"); getAppDriver().findElement(MobileBy.custom("ai:search")).click();
Capabilities set capabilities.setCapability("customFindModules", customFindModules); capabilities.setCapability("testaiFindMode", "element_lookup"); capabilities.setCapability("testaiObjectDetectionThreshold", "0.1"); capabilities.setCapability("shouldUseCompactResponses", false);
2020-01-29 13:42:08,349 120921 [TestNG-tests-1] INFO com.walmart.asda.mobile.poc.TestSearch_01 - Executing : findElementByCustom :::----> Search icon using AI [HTTP] --> POST /wd/hub/session/8653e25f-6cd0-4603-b4c2-a942bf0c2f77/element [HTTP] {"using":"-custom","value":"ai:search"} [debug] [W3C (8653e25f)] Calling AppiumDriver.findElement() with args: ["-custom","ai:search","8653e25f-6cd0-4603-b4c2-a942bf0c2f77"] [debug] [XCUITest] Executing command 'findElement' [debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier' 2020-01-29 13:42:10.518184: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-01-29 13:42:10.544574: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x10c8c5770 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-01-29 13:42:10.544601: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version [debug] [BaseDriver] Waiting up to 0 ms for condition [ai] Will use object detection method for finding elements via AI [ai] Setting confidence threshold to default value of 0.1 [ai] Getting window size in case we need to scale screenshot [debug] [WD Proxy] Proxying [GET /window/size] to [GET http://localhost:46146/session/27188DFA-B718-41AD-AEF5-B4FCBAFDDBB4/window/size] with no body [debug] [WD Proxy] Got response with status 200: "{\n \"value\" : {\n \"width\" : 375,\n \"height\" : 812\n },\n \"sessionId\" : \"27188DFA-B718-41AD-AEF5-B4FCBAFDDBB4\",\n \"status\" : 0\n}" [ai] Getting screenshot to use for classifier [debug] [XCUITest] Taking screenshot with WDA [debug] [WD Proxy] Matched '/screenshot' to command name 'getScreenshot' [debug] [WD Proxy] Proxying [GET /screenshot] to [GET http://localhost:46146/session/27188DFA-B718-41AD-AEF5-B4FCBAFDDBB4/screenshot] with no body [debug] [WD Proxy] Got response with status 200: "{\n \"value\" : \"iVBORw0KGgoAAAANSUhEUgAABGUAAAmECAIAAAAjR0TuAAAAAXNSR0IArs4c6QAA\r\nABxpRE9UAAAAAgAAAAAAAATCAAAAKAAABMIAAATCAAavbYrc7aAAAEAASURBVHgB\r\n7J0HnB5Vvfdf3\/fe973FexWlpSebTSOEIiBdRK4FFJAqoleqIioipGzfzaaRAAFU\r\nVNSLeilepQTStj9bEpr03qtECISQutnd7G7m\/f7P\/3lOnuxuQgrJRvKbz2Ry5syZ\r\nc\/7znTnP\/n9zyvyv\/6VFBERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERAB... [ai] Turning screenshot into HTML image for use with canvas [ai] Screenshot and screen size did not match. Screen size is 375x812 but screenshot size is 1125x2436. Scaled screenshot to match screen size [ai] Writing screenshot to /usr/local/lib/node_modules/test-ai-classifier/build-js/lib/Wed, 29 Jan 2020 08:12:11 GMT_classifiertest.png for use in object detection [ai] Gathering object candidates from tensorflow native 2020-01-29 13:42:11.456773: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /usr/local/lib/node_modules/test-ai-classifier/model/obj_detection_model 2020-01-29 13:42:11.756360: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve } 2020-01-29 13:42:12.260225: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle. 2020-01-29 13:42:12.261404: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: /usr/local/lib/node_modules/test-ai-classifier/model/obj_detection_model/variables/variables.index 2020-01-29 13:42:12.261778: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 805000 microseconds. [ai] Candidates retrieved [ai] Making label predictions based on element images This model execution did not contain any nodes with control flow or dynamic output shapes. You can use model.execute() instead. [ai] Predictions for 34 element(s) took 0.686 seconds [ai] Found 1 matching elements [ai] Highest confidence of any element for desired label 'search' was 0.9994727373123169 [ai] Registering image elements with driver [debug] [W3C (8653e25f)] Encountered internal error running command: TypeError: driver.registerImageElement is not a function [debug] [W3C (8653e25f)] at map (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:365:19) [debug] [W3C (8653e25f)] at Array.map ()
[debug] [W3C (8653e25f)] at findByObjectDetector (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:357:24)
[debug] [W3C (8653e25f)] at Object.find (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:375:12)
[debug] [W3C (8653e25f)] at condition (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/find.js:159:16)
[debug] [W3C (8653e25f)] at wrappedCondFn (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/timeout.js:157:12)
[HTTP] <-- POST /wd/hub/session/8653e25f-6cd0-4603-b4c2-a942bf0c2f77/element 500 17780 ms - 587
[HTTP]
org.openqa.selenium.WebDriverException: An unknown server-side error occurred while processing the command. Original error: driver.registerImageElement is not a function Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03' System info: host: 'm-c02xf25rjg5h', ip: 'fe80:0:0:0:1c5b:4ba:7301:ce39%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '1.8.0_192' Driver info: io.appium.java_client.ios.IOSDriver Capabilities {app: /Users/kverma1/Documents/mo..., automationName: XCUITest, browserName: , customFindModules: {ai: test-ai-classifier}, databaseEnabled: false, deviceName: iPhone X, fullReset: true, javascriptEnabled: true, locationContextEnabled: false, networkConnectionEnabled: false, newCommandTimeout: 120, noReset: false, platform: MAC, platformName: iOS, platformVersion: 12.0, shouldUseCompactResponses: false, takesScreenshot: true, testaiFindMode: object_detection, testaiObjectDetectionThreshold: 0.4, udid: E18161C4-00ED-481A-9281-867..., useNewWDA: false, wdaLocalPort: 46146, webStorageEnabled: false} Session ID: 8653e25f-6cd0-4603-b4c2-a942bf0c2f77 *** Element info: {Using=-custom, value=ai:search}
`