Open ericsavoiewaltz opened 6 years ago
jlipps
commented
6 years ago It looks like the issue is that the connection to WDA times out while the classifier is running. I would therefore recommend exploring the XCUITest driver docs to discover whether there is a capability that can modulate that timeout. At any rate nothing in this log looks like a bug in this plugin, so closing here.
jlipps
commented
6 years ago From https://github.com/appium/appium-xcuitest-driver it looks like wdaConnectionTimeout could be increased. If you're still running into issues, please provide a full log at the Appium issue tracker.
ericsavoiewaltz
commented
6 years ago Cool thanks
ericsavoiewaltz
commented
6 years ago @jlipps
I have some appium output here that shows that the WDA losing connection doesn't impact the custom plugin's ability to find an element. You can see it lose connection and find the QR code with a confidence interval of 0.9+ and then fail to find the send icon with a confidence interval of 0.3+. The default threshold is at 0.2.
I don't think this is just solved by the time out. Can you please take another look?
[debug] [W3C (db475602)] Calling AppiumDriver.findElement() with args: ["-custom","ai:qr code","db475602-5ad2-4bd1-8865-c0d8c53a8ca9"]
[debug] [XCUITest] Executing command 'findElement'
[debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier'
[debug] [BaseDriver] Waiting up to 20000 ms for condition
[ai] Retrieving current settings to check element response attributes
[ai] We will need to update settings to include element response attributes
[debug] [JSONWP Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [JSONWP Proxy] Proxying [POST /appium/settings] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/appium/settings] with body: {"settings":{"elementResponseAttributes":"rect"}}
[debug] [JSONWP Proxy] Got response with status 200: {"value":{"elementResponseAttributes":"rect","shouldUseCompactResponses":false},"sessionId":"3C65A71B-BE78-4150-8321-40AC8549680E","status":0}
[ai] Setting confidence threshold to default value of 0.2
[ai] Retrieving data for all leaf-node elements on screen
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id
[debug] [BaseDriver] Waiting up to 20000 ms for condition
[debug] [JSONWP Proxy] Matched '/elements' to command name 'findElements'
[debug] [JSONWP Proxy] Proxying [POST /elements] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/elements] with body: {"using":"xpath","value":"//*[not(child::*)]"}
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [JSONWP Proxy] Got response with status 200: {"value":[{"ELEMENT":"84000000-0000-0000-7401-000000000000","rect":{"y":20,"x":8,"width":55,"height":44}},{"ELEMENT":"85000000-0000-0000-7401-000000000000","rect":{"y":26,"x":125,"width":125,"height":32}},{"ELEMENT":"AB000000-0000-0000-7401-000000000000","rect":{"y":80,"x":-279,"width":50,"height":50}},{"ELEMENT":"AC000000-0000-0000-7401-000000000000","rect":{"y":80,"x":-213,"width":197,"height":21}},{"ELEMENT":"AD000000-0000-0000-7401-000000000000","rect":{"y":108,"x":-213,"width":197,"height":22}},{"ELEMENT":"B1000000-0000-0000-7401-000000000000","rect":{"y":205,"x":-295,"width":295,"height":1}},{"ELEMENT":"B0000000-0000-0000-7401-000000000000","rect":{"y":157,"x":-280,"width":36,"height":36}},{"ELEMENT":"B2000000-0000-0000-7401-000000000000","rect":{"y":146,"x":-229,"width":214,"height":60}},{"ELEMENT":"B5000000-0000-0000-7401-000000000000","rect":{"y":265,"x":-295,"width":295,"height":1}},{"ELEMENT":"B4000000-0000-0000-7401-000000000000","rect":{"y":217,"x":-280,"width":36,"height":36}},{"ELEMENT":"B60...
[ai] Getting window size in case we need to scale screenshot
[ai] Getting screenshot to use for classifier
[debug] [XCUITest] Taking screenshot with WDA
[debug] [JSONWP Proxy] Matched '/screenshot' to command name 'getScreenshot'
[debug] [JSONWP Proxy] Proxying [GET /screenshot] to [GET http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/screenshot] with no body
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [JSONWP Proxy] Got response with status 200: "{\n \"value\" : \"iVBORw0KGgoAAAANSUhEUgAAAu4AAAU2CAIAAABFtaRRAAAAAXNSR0IArs4c6QAA\\r\\nABxpRE9UAAAAAgAAAAAAAAKbAAAAKAAAApsAAAKbAAGayJ52Y+gAAEAASURBVHgB\\r\\n7L1ldFzn1f7txrYYLJOYjGIyyJY00ohli1myLIvB0LRp0zA1aZq0aZ+AHXLiGGJm\\r\\nlGUxMzMzWmZ2DPd77fuMFCVpHft51v\\/LG2XtddY9Z86AlZlzfnPta+89TSSVIZqZ\\r\\nJZqZJ5pRIJqZQyGdK5LOtJM+bydzTiSTKpLOFs0sE82sEPaLp1eKZ5TgAIStVBnC\\r\\nXibHQSbLfmaReHqZ44xKx5mluOkgk24nm4lwkM77WYhl8hHCTmE9eesgm+sgU+gg\\r\\nXUUhm+8gl+Ugl4EQy2dSKFwQy6eL5fLEsqVimWqxdJ1IrkQsV+Aim+4uk+4uneMq\\r\\nVYS77OVK7OWzHeUyXWTy3GeWuElVuMhU2CsU2yrlu8gWusjlO8nnIhzlih3lSh3k\\r\\nKcQKZWKFEieFAieFPGf5AoSrfC4PLBB548FvypU4y5U5y5U7y1XSVr4Ex7so5rgo\\r\\nZTnJ1TrJVboolLooFLoo5mPrJl\\/qJldOIV\\/qrJjvrJTnqJTvqFjsqFTpqFjrMivD\\r\\nZVami3K+i1KRi1KJi2IZbZWKXJXyXZVyXZWzaYu1YpG7Qpm7fIWrYgnCTan0V8Nd\\r\\nuQwx+TBhz9NvPWaVP0WUesySxBqVsmeKtbPLJ0J44MTNJyw851QghAOE9ZO3XnMr\\r\\nEcIxwvrJW+95Vf\\/H8Jlf\\/avhq1rzfww\\/tVoEf5I6P7VGHrV+6lW+GmUIP\\/UKfkCj\\r\\nn2obxfwOf7VWhK9aI8JPowHhr17nr45tk79ai596C+3UrPTTKvXTKsLWX6PSX6NG\\r\\nOCZIvTl...
[ai] Turning screenshot into HTML image for use with canvas
[ai] Screenshot and screen size did not match. Screen size is 375x667 but screenshot size is 750x1334. Scaling screenshot to match screen size
[ai] Getting screenshot slices for each element
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[ai] Making label predictions based on element images
[ai] Found 1 matching elements
[ai] Highest confidence of any element for desired label 'qr code' was 0.9949374198913574
[ai] Resetting element response attribute setting to original value: "type,label"
[debug] [JSONWP Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [JSONWP Proxy] Proxying [POST /appium/settings] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/appium/settings] with body: {"settings":{"elementResponseAttributes":"type,label"}}
[debug] [JSONWP Proxy] Got response with status 200: {"value":{"elementResponseAttributes":"type,label","shouldUseCompactResponses":false},"sessionId":"3C65A71B-BE78-4150-8321-40AC8549680E","status":0}
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [W3C (db475602)] Responding to client with driver.findElement() result: {"element-6066-11e4-a52e-4f735466cecf":"CD000000-0000-0000-7401-000000000000","rect":{"y":106,"x":31,"width":313,"height":312}}
[HTTP] <-- POST /wd/hub/session/db475602-5ad2-4bd1-8865-c0d8c53a8ca9/element 200 49155 ms - 137
[HTTP]
[HTTP] --> POST /wd/hub/session/db475602-5ad2-4bd1-8865-c0d8c53a8ca9/element
[HTTP] {"using":"-custom","value":"ai:send"}
[debug] [W3C (db475602)] Calling AppiumDriver.findElement() with args: ["-custom","ai:send","db475602-5ad2-4bd1-8865-c0d8c53a8ca9"]
[debug] [XCUITest] Executing command 'findElement'
[debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier'
[debug] [BaseDriver] Waiting up to 20000 ms for condition
[ai] Retrieving current settings to check element response attributes
[ai] We will need to update settings to include element response attributes
[debug] [JSONWP Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [JSONWP Proxy] Proxying [POST /appium/settings] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/appium/settings] with body: {"settings":{"elementResponseAttributes":"rect"}}
[debug] [JSONWP Proxy] Got response with status 200: {"value":{"elementResponseAttributes":"rect","shouldUseCompactResponses":false},"sessionId":"3C65A71B-BE78-4150-8321-40AC8549680E","status":0}
[ai] Setting confidence threshold to default value of 0.2
[ai] Retrieving data for all leaf-node elements on screen
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id
[debug] [BaseDriver] Waiting up to 20000 ms for condition
[debug] [JSONWP Proxy] Matched '/elements' to command name 'findElements'
[debug] [JSONWP Proxy] Proxying [POST /elements] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/elements] with body: {"using":"xpath","value":"//*[not(child::*)]"}
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [JSONWP Proxy] Got response with status 200: {"value":[{"ELEMENT":"84000000-0000-0000-7401-000000000000","rect":{"y":20,"x":8,"width":55,"height":44}},{"ELEMENT":"85000000-0000-0000-7401-000000000000","rect":{"y":26,"x":125,"width":125,"height":32}},{"ELEMENT":"CD000000-0000-0000-7401-000000000000","rect":{"y":106,"x":31,"width":313,"height":312}},{"ELEMENT":"CE000000-0000-0000-7401-000000000000","rect":{"y":468,"x":26,"width":323,"height":41}},{"ELEMENT":"D0000000-0000-0000-7401-000000000000","rect":{"y":517,"x":116,"width":143,"height":142}},{"ELEMENT":"D8000000-0000-0000-7401-000000000000","rect":{"y":64,"x":0,"width":375,"height":603}},{"ELEMENT":"AB000000-0000-0000-7401-000000000000","rect":{"y":80,"x":16,"width":50,"height":50}},{"ELEMENT":"AC000000-0000-0000-7401-000000000000","rect":{"y":80,"x":82,"width":197,"height":21}},{"ELEMENT":"AD000000-0000-0000-7401-000000000000","rect":{"y":108,"x":82,"width":197,"height":22}},{"ELEMENT":"B1000000-0000-0000-7401-000000000000","rect":{"y":205,"x":0,"width":295,"height":1}},{"ELEMENT":"B0000000-0000-0...
[ai] Getting window size in case we need to scale screenshot
[ai] Getting screenshot to use for classifier
[debug] [XCUITest] Taking screenshot with WDA
[debug] [JSONWP Proxy] Matched '/screenshot' to command name 'getScreenshot'
[debug] [JSONWP Proxy] Proxying [GET /screenshot] to [GET http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/screenshot] with no body
[debug] [JSONWP Proxy] Got response with status 200: "{\n \"value\" : \"iVBORw0KGgoAAAANSUhEUgAAAu4AAAU2CAIAAABFtaRRAAAAAXNSR0IArs4c6QAA\\r\\nABxpRE9UAAAAAgAAAAAAAAKbAAAAKAAAApsAAAKbAAL\\/ghKz6t0AAEAASURBVHgB\\r\\n7L11eFTn1v7PKRAXgsUFjxsSSDLJxEPchRDiArQ9Pdaeek\\/l9HgPhRotUCjuGkLc\\r\\n3d1dCe7Fnt+9nj1JU6Pwvtf7\\/ePXcK3rufbs2ZmZhJm9P3Ove601TSSVIZqZJZqZ\\r\\nJ5pRIJqZQyGdK5LOtJU+bytzTiSTKpLOFs0sE82sEPaLp1eKZ5TgAISNVBnCTibH\\r\\nXibLbmaReHqZw4xKh5mluGkvk24rm4mwl877QYhl8hHCTmF78movm2svU2gvXUUh\\r\\nm28vl2Uvl4EQy2dSKFwQy6eL5fLEsqVimWqxdJ1IrkQsV+Asm+4mk+4mneMiVYS7\\r\\n7ORK7OSzHeQynWXy3GaWuEpVOMtU2CkU2yjlO8sWOsvlO8rnIhzkih3kSu3lKcQK\\r\\nZWKFEkeFAkeFPCf5AoSLfC4PbCDyxoPflCtxkitzkit3kqukVb4Exzsr5jgrZTnK\\r\\n1TrKVTorlDorFDor5mN1lS91lSunkC91Usx3UspzUMp3UCx2UKp0UKx1npXhPCvT\\r\\nWTnfWanIWanEWbGMVqUiF6V8F6VcF+VsWrGtWOSmUOYmX+GiWIJwVSr9xXBTLkNM\\r\\nPkzY8\\/Sr+6zyp4hS91mSWKtS9kzhMbt8IoQfnLj5hA3PORUI4QBh+8mr19xKhHCM\\r\\nsP3k1Xte1f8yfOZX\\/2L4qtb8L8NPrRbBH6TOT7WFQq2RdqpX8aj+blujDHv81ev8\\r\\n1Zv85vX6zev2V2vFtr9GtSRwl1oL7cSK\\/eoNdLBGDcIPP6LREKBRjwhSbwxSbw5W\\r\\naw5Wrw\\/...
[ai] Turning screenshot into HTML image for use with canvas
[ai] Screenshot and screen size did not match. Screen size is 375x667 but screenshot size is 750x1334. Scaling screenshot to match screen size
[ai] Getting screenshot slices for each element
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[ai] Making label predictions based on element images
[ai] Found 0 matching elements
[ai] Highest confidence of any element for desired label 'send' was 0.3476186692714691
[ai] Resetting element response attribute setting to original value: "type,label"@ericsavoiewaltz I'd need to see the full log to comment any further. This log doesn't show the response to the client, for example.
ericsavoiewaltz
commented
6 years ago jlipps
commented
6 years ago OK, thanks, will reopen and investigate more when I have a bit of time.
jlipps
commented
5 years ago @ericsavoiewaltz can you retry with v 2.1 of the classifier plugin, and let me know if this is still happening?
pramodKarande
commented
5 years ago @cnavrides @jlipps while training , do we need to add images in specific resolution and format??
Appium Output: Unless I completely misunderstand how this works I think this is a bug?
[ai] Setting confidence threshold to default value of 0.2[ai] Retrieving data for all leaf-node elements on screen [debug] [BaseDriver] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id [debug] [BaseDriver] Waiting up to 20000 ms for condition [debug] [JSONWP Proxy] Matched '/elements' to command name 'findElements' [debug] [JSONWP Proxy] Proxying [POST /elements] to [POST http://localhost:8001/session/A4853193-80C9-4FB2-BF37-9C4E9A94633D/elements] with body: {"using":"xpath","value":"//[not(child::)]"} [debug] [JSONWP Proxy] Got response with status 200: {"value":[{"ELEMENT":"7E000000-0000-0000-0614-000000000000","rect":{"y":44,"x":8,"width":51,"height":44}},{"ELEMENT":"7F000000-0000-0000-0614-000000000000","rect":{"y":52,"x":132,"width":111,"height":28}},{"ELEMENT":"B3000000-0000-0000-0614-000000000000","rect":{"y":200,"x":41,"width":293,"height":294}},{"ELEMENT":"B4000000-0000-0000-0614-000000000000","rect":{"y":613,"x":26,"width":323,"height":41}},{"ELEMENT":"B6000000-0000-0000-0614-000000000000","rect":{"y":662,"x":116,"width":143,"height":142}},{"ELEMENT":"84000000-0000-0000-0614-000000000000","rect":{"y":88,"x":0,"width":375,"height":724}},{"ELEMENT":"92000000-0000-0000-0614-000000000000","rect":{"y":104,"x":16,"width":50,"height":50}},{"ELEMENT":"93000000-0000-0000-0614-000000000000","rect":{"y":104,"x":82,"width":197,"height":21}},{"ELEMENT":"94000000-0000-0000-0614-000000000000","rect":{"y":132,"x":82,"width":197,"height":22}},{"ELEMENT":"99000000-0000-0000-0614-000000000000","rect":{"y":229,"x":0,"width":295,"height":1}},{"ELEMENT":"98000000-0000... [ai] Getting window size in case we need to scale screenshot [ai] Getting screenshot to use for classifier [debug] [XCUITest] Taking screenshot with WDA [debug] [JSONWP Proxy] Matched '/screenshot' to command name 'getScreenshot' [debug] [JSONWP Proxy] Proxying [GET /screenshot] to [GET http://localhost:8001/session/A4853193-80C9-4FB2-BF37-9C4E9A94633D/screenshot] with no body [debug] [XCUITest] Connection to WDA timed out [debug] [iProxy] recv failed: Operation not permitted [debug] [JSONWP Proxy] Got response with status 200: "{\n \"value\" : \"iVBORw0KGgoAAAANSUhEUgAABGUAAAmECAIAAAAjR0TuAAAAAXNSR0IArs4c6QAA\r\nABxpRE9UAAAAAgAAAAAAAATCAAAAKAAABMIAAATCAATtHGwVVeMAAEAASURBVHgB\r\n7N0HmF3HdSBoe9ZiAJiBzrkbzNbY9IyTSCIwgEDnHABKthVsU\/Ptjndnvf7Wnm8c\r\n1mPLsvcbT9ixPfNJligmBYpiBEmAiI3QaKDROSEwyxItUaIogsh3T916\/XDw6lW9\r\nU6fulZrE4dY8V586dbtfg1q+H1V16mfu\/Mj2vG3lJTsSaasu3ZlIW33ZrsTbmssH\r\ns00\/PPslu3PXkt2JtLuX7km83XPF3gTbvVfug6YfqPshr2uvGkq83Xf1fnZbd81w\r\nQm1o3TUXtPXX7oeGgzrCe228bjiR1rTsQCKtefnBRFpL0UiqrbX4UCKtrWQ0kdZe\r\nOpZI6ygbT7V1lk8k0roqJhNp3ZVTibSequm8TT8875BXsLd6JpHWVzObSOuvnUu8\r\nDdTNJ9421B9OpG1sOJJIu3\/F0bxNPzzvUN7gx68\/9hNrn7jhZXb7jRtfSaT95k2v\r\nJt5+6+bXsk0\/PPslu\/PJW15PpH3q1jcSb5\/++TfZ7TMf\/Xaq7bf\/5T8l0n7nF76T\r\nSPvdX\/xu3qYfnneIEvyZvFiCYCJYgockgiV4SOJYggdiFImXfCklXqKB6gIsAZPE\r\nSxRQpYoleHgiWIKHJIIleEgiWIKHpIoleHgiWIKHJIIleEgiWIKH2PAjXqKzKnEs\r\nwQMTwRI8JBEswUPy4geC4iU3qxLHEjwQo0i85KZUqliChyeCJXhIIliCh9jwI17i\r\nrzuJl3yNhPPFS+Ilc0mKYiFKjniJJyjxEk9Q4iW6i2yZ4iWbpsz4T2xxCb4Re3EJ\r... [ai] Turning screenshot into HTML image for use with canvas [ai] Screenshot and screen size did not match. Screen size is 375x812 but screenshot size is 1125x2436. Scaling screenshot to match screen size [ai] Getting screenshot slices for each element [debug] [XCUITest] Connection to WDA timed out [debug] [iProxy] recv failed: Operation not permitted [ai] Making label predictions based on element images