search

KazuCocoa / appium_dart

Appium client in Dart based on webdriver.dart
https://pub.dev/packages/appium_driver
Apache License 2.0
45 stars 14 forks source link

How to perform touch operations in dart for iOS outside flutter application? #48

Open ferozk opened 1 year ago

ferozk commented 1 year ago

Is there any way to open notification center and control center on iOS in dart? I saw perform/touch or swipe operations are not implemented in the package. Is it possible to implement these?

[ ] [HttpMethod.httpPost, 'session/:session_id/touch/perform'] # W3C actions should be an alternative
[ ] [HttpMethod.httpPost, 'session/:session_id/touch/multi/perform'] # W3C actions should be an alternative

How to perform above commands in dart for iOS devices? Need for opening control center and notification center

I tried following commands it is getting executed but nothing happens. I changed the context to NATIVE before executing following commands await driver.execute("mobile: dragFromToForDuration", [{"duration": 3.0,"fromX":9, "fromY":5, "toX":9,"toY": 700}]); await driver.execute('mobile: scroll', [{'direction': 'up'}]); - this scroll inside app

With java client I am able to do on same device using following so wanted the similar operation in dart

new TouchAction(driver)
        .press(PointOption.point(0, 0))
        .waitAction(new WaitOptions().withDuration(Duration.ofMillis(1000)))
        .moveTo(PointOption.point(0, 500))
        .release()
        .perform();

Appium - 1.22.3 Flutter driver - 0.0.36

On android I was able to open notifications center using following command await driver.execute("mobile: dragGesture", [{"startX":0, "startY":0, "endX":0,"endY": 500}]);

KazuCocoa commented 1 year ago

Perhaps https://github.com/google/webdriver.dart/blob/ff5ccb1522edf4bed578ead4d65e0cbc1f2c4f02/lib/src/handler/w3c_handler.dart#L28-L31 are W3C actions in Dart.

https://pub.dev/documentation/webdriver/latest/webdriver.io/Mouse-class.html https://pub.dev/documentation/webdriver/latest/webdriver.io/Keyboard-class.html

I tried following commands it is getting executed but nothing happens

Then, the vanilla XCTest framework by Apple was able to behave so. Potentially via W3C action also behaves the same

ferozk commented 1 year ago

https://pub.dev/documentation/webdriver/latest/webdriver.io/Mouse-class.html

Seem this is not supported by flutter driver we can use with webdriver.io only for that we need to create webdriver object and to make that work. I need using appium driver or flutter driver.

With Flutter driver tried following 

      await driver.webdriver.mouse.moveTo(xOffset: 0, yOffset: 0);
      await driver.webdriver.mouse.down();
      await driver.webdriver.mouse.moveTo(xOffset: 0, yOffset: 500);
      await driver.webdriver.mouse.up();

It is giving following error

Unsupported operation: VMServiceFlutterDriver does not support webDriver

package:flutter_driver/src/driver/vmservice_driver.dart 278:39  VMServiceFlutterDriver.webDriver

I tried to use with appium driver object as following 

      await driver.mouse.moveTo(xOffset: 0, yOffset: 0);
      await driver.mouse.down();
      await driver.mouse.moveTo(xOffset: 0, yOffset: 500);
      await driver.mouse.up();

This gives following error on appium sever log

[HTTP] --> POST /wd/hub/session/c2566928-5940-4c99-a14e-5081b4c4ff3f/context
[HTTP] {"name":"NATIVE_APP"}
[debug] [W3C (c2566928)] Calling AppiumDriver.setContext() with args: ["NATIVE_APP","c2566928-5940-4c99-a14e-5081b4c4ff3f"]
[debug] [FlutterDriver] Executing Flutter driver command 'setContext'
[debug] [W3C (c2566928)] Responding to client with driver.setContext() result: null
[HTTP] <-- POST /wd/hub/session/c2566928-5940-4c99-a14e-5081b4c4ff3f/context 200 1 ms - 14
[HTTP] 
[HTTP] --> POST /wd/hub/session/c2566928-5940-4c99-a14e-5081b4c4ff3f/actions
[HTTP] {"actions":[{"type":"pointer","id":"mouses","actions":[{"type":"pointerMove","origin":"pointer","x":0,"y":0}]}]}
[debug] [W3C (c2566928)] Calling AppiumDriver.performActions() with args: [[{"type":"pointer","id":"mouses","actions":[{"type":"pointerMove","origin":"pointer","x":0,"y":0}]}],"c2566928-5940-4c99-a14e-5081b4c4ff3f"]
[debug] [FlutterDriver] Executing proxied driver command 'performActions'
[debug] [UiAutomator2] Received the following W3C actions: [
[debug] [UiAutomator2]   {
[debug] [UiAutomator2]     "type": "pointer",
[debug] [UiAutomator2]     "id": "mouses",
[debug] [UiAutomator2]     "actions": [
[debug] [UiAutomator2]       {
[debug] [UiAutomator2]         "type": "pointerMove",
[debug] [UiAutomator2]         "origin": "pointer",
[debug] [UiAutomator2]         "x": 0,
[debug] [UiAutomator2]         "y": 0
[debug] [UiAutomator2]       }
[debug] [UiAutomator2]     ]
[debug] [UiAutomator2]   }
[debug] [UiAutomator2] ]
[debug] [UiAutomator2] Preprocessed actions: [
[debug] [UiAutomator2]   {
[debug] [UiAutomator2]     "type": "pointer",
[debug] [UiAutomator2]     "id": "mouses",
[debug] [UiAutomator2]     "actions": [
[debug] [UiAutomator2]       {
[debug] [UiAutomator2]         "type": "pointerMove",
[debug] [UiAutomator2]         "origin": "pointer",
[debug] [UiAutomator2]         "x": 0,
[debug] [UiAutomator2]         "y": 0
[debug] [UiAutomator2]       }
[debug] [UiAutomator2]     ],
[debug] [UiAutomator2]     "parameters": {
[debug] [UiAutomator2]       "pointerType": "touch"
[debug] [UiAutomator2]     }
[debug] [UiAutomator2]   }
[debug] [UiAutomator2] ]
[debug] [WD Proxy] Matched '/actions' to command name 'performActions'
[debug] [WD Proxy] Proxying [POST /actions] to [POST http://127.0.0.1:8203/wd/hub/session/5738a13e-0704-4c65-9767-2ddfef65469e/actions] with body: {"actions":[{"type":"pointer","id":"mouses","actions":[{"type":"pointerMove","origin":"pointer","x":0,"y":0}]}]}
[WD Proxy] Got response with status 400: {"sessionId":"5738a13e-0704-4c65-9767-2ddfef65469e","value":{"error":"invalid argument","message":"Exception while reading JSON","stacktrace":"io.appium.uiautomator2.common.exceptions.InvalidArgumentException: Exception while reading JSON\n\tat io.appium.uiautomator2.handler.W3CActions.safeHandle(W3CActions.java:92)\n\tat io.appium.uiautomator2.handler.request.SafeRequestHandler.handle(SafeRequestHandler.java:59)\n\tat io.appium.uiautomator2.server.AppiumServlet.handleRequest(AppiumServlet.java:266)\n\tat io.appium.uiautomator2.server.AppiumServlet.handleHttpRequest(AppiumServlet.java:260)\n\tat io.appium.uiautomator2.http.ServerHandler.channelRead(ServerHandler.java:68)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)\n\tat io.n...
[debug] [W3C] Matched W3C error code 'invalid argument' to InvalidArgumentError
[debug] [W3C (c2566928)] Encountered internal error running command: io.appium.uiautomator2.common.exceptions.InvalidArgumentException: Exception while reading JSON
[debug] [W3C (c2566928)]    at io.appium.uiautomator2.handler.W3CActions.safeHandle(W3CActions.java:92)
[debug] [W3C (c2566928)]    at io.appium.uiautomator2.handler.request.SafeRequestHandler.handle(SafeRequestHandler.java:59)
[debug] [W3C (c2566928)]    at io.appium.uiautomator2.server.AppiumServlet.handleRequest(AppiumServlet.java:266)
[debug] [W3C (c2566928)]    at io.appium.uiautomator2.server.AppiumServlet.handleHttpRequest(AppiumServlet.java:260)
[debug] [W3C (c2566928)]    at io.appium.uiautomator2.http.ServerHandler.channelRead(ServerHandler.java:68)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
[debug] [W3C (c2566928)]    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
[debug] [W3C (c2566928)]    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:435)
[debug] [W3C (c2566928)]    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
[debug] [W3C (c2566928)]    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
[debug] [W3C (c2566928)]    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
[debug] [W3C (c2566928)]    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
[debug] [W3C (c2566928)]    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[debug] [W3C (c2566928)]    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
[debug] [W3C (c2566928)]    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
[debug] [W3C (c2566928)]    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
[debug] [W3C (c2566928)]    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:611)
[debug] [W3C (c2566928)]    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:552)
[debug] [W3C (c2566928)]    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:466)
[debug] [W3C (c2566928)]    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438)
[debug] [W3C (c2566928)]    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
[debug] [W3C (c2566928)]    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[debug] [W3C (c2566928)]    at java.lang.Thread.run(Thread.java:923)
[debug] [W3C (c2566928)] Caused by: io.appium.uiautomator2.utils.w3c.ActionsParseException: Missing duration key for action item 'io.appium.uiautomator2.model.api.touch.w3c.W3CGestureModel@f6da9c2' of action with id 'mouses'
[debug] [W3C (c2566928)]    at io.appium.uiautomator2.utils.w3c.ActionsTokenizer.extractDuration(ActionsTokenizer.java:116)
[debug] [W3C (c2566928)]    at io.appium.uiautomator2.utils.w3c.ActionsTokenizer.applyPointerActionToEventsMapping(ActionsTokenizer.java:427)
[debug] [W3C (c2566928)]    at io.appium.uiautomator2.utils.w3c.ActionsTokenizer.tokenize(ActionsTokenizer.java:517)
[debug] [W3C (c2566928)]    at io.appium.uiautomator2.handler.W3CActions.safeHandle(W3CActions.java:83)
[debug] [W3C (c2566928)]    ... 34 more
[debug] [W3C (c2566928)] 
[HTTP] <-- POST /wd/hub/session/c2566928-5940-4c99-a14e-5081b4c4ff3f/actions 400 47 ms - 4280

Seems it is trying to send only first command for moveTo at a time.

Is there any sample code in dart using appium dart package similar to Java client where we send multiple touch operations in same request as follow 

new TouchAction(driver)
        .press(PointOption.point(0, 0))
        .waitAction(new WaitOptions().withDuration(Duration.ofMillis(1000)))
        .moveTo(PointOption.point(0, 500))
        .release()
        .perform();
KazuCocoa commented 1 year ago

Maybe the original Dart WebDriver client did not fully implement the W3C actions. Let me extend this library when I have time. (mouse and keyboard are from the original Dart WebDriver in this repo)

The TouchAction endpoint will be removed from Appium as non-W3C spec (the official clients currently print warnings), so this client won't have the endpoint

santkond commented 1 year ago

Hi @KazuCocoa , I am facing the same issue as ferozk has mentioned. Unable to open notification center in IOS device. Tried the same options but no luck. Any Approx date by when it can be fixed.