Not able to scroll using TouchScroll

[padmaja2812]

Hi, I am trying to use TouchScroll. Only mouse location is changing according to the coordinates but it not holding left click and scrolling. Can you help to resolve the issue.

Thanks Padmaja

[NickAb]

Could you please provide some extra details:

1. How do you call TouchScroll, what values do you use for offset?
2. How much of the element you are trying to scroll is on screen? Or are you scrolling without element (i.e. form centre of the screen)?
3. Have you tried TouchFlick, does it work?
4. How do you check that coordinates have changed?

[padmaja2812]

-With command executor we call touchscroll and values are Offset Values {X = 240 Y = 427} {X = 240 Y = 335 {X = 240 Y = 242} {X = 240 Y = 150} {X = 240 Y = 57} {X = 240 Y = -35} {X = 240 Y = 28} -No we are just trying to scroll to get element in view port. -No we haven't tried touch flick but will try with that as well. -the position of cursor changes after every leftdown().

[NickAb]

Can you clarify what cursor are talking about? Driver should not be able to move the cursor of your host system (arrow pointer) as it sends mouse actions directly to virtual machine corresponding to the emulator you are using.

[padmaja2812]

cursor means the arrow pointer.

[NickAb]

Cursor pointer from host OS or some kind of cursor pointer in your app?

What version of the driver do you use? Winium.StoreApps.Driver.exe --version

[padmaja2812]

Cursor i m referencing to is the mouse pointer from host OS. I am running outer server from code (VS) in debug mode.

Thanks Padmaja

[NickAb]

You host OS pointer should not be moving anywhere, because driver sends mouse events directly to emulator virtual machine. Could you please make a screen capture of what is happening? I can not quite understand how can it be that your host OS cursor is moving.

As a workaround you can find a scrollable container element that is a parent of the element you want to scroll into the view. And then use ExecuteScript command with automation: ScrollPattern.Scroll, the scrollable container element, {"v": "smallIncrement", "count": 10}. Here is an example in python, we will be scrolling some listbox that contains the item we want to scroll into the view:

list_box = self.driver.find_element_by_id('MyListBox')
self.driver.execute_script('automation: ScrollPattern.Scroll', list_box, {"v": "smallIncrement", "count": 10})

More info on ExecuteScript can be found in wiki page

[padmaja2812]

It's not the OS pointer moving, only the pointer in the virtual machine/emulator is moving and when I am trying to scroll the element that pointer is only moving in the scroll direction and not scrolling.

[NickAb]

Are you trying to scroll some custom element? Have you tried ExecuteScript with ScrollPattern?

[padmaja2812]

I am trying to scroll the current screen but it is not scrolling at all, as you can see in the recording shared. The reason, I see is that we are not using touch events but are using mouse events which don't perform scrolling in Windows 10 mobile. https://youtu.be/Reh6AugXrNI

[NickAb]

Please try using ExecuteScript command with automation: ScrollPattern.Scroll as a wrokaround, here is example in python:

listbox = self.driver.find_element_by_id("MyListBox")
        scroll_info = {"v": "smallIncrement", "count": 10}
        self.driver.execute_script("automation: ScrollPattern.Scroll", listbox, scroll_info)

It seems that on Windows 10 any mouse action (movement) we send releases previously hold left button. This issues seems to be affecting Windows 10 only. It might take some time to find another solution for touch simulation.

Sorry it took so long, please let me know if ExecuteScript workaround works for you.

[padmaja2812]

Hi, Apologies for replying so late. This approach is not working for custom controls. "Element does not support IScrollProvider control pattern interface." do we need to implement this with each custom control?

Thanks Padmaja

[NickAb]

Make sure that you are passing a scrollable container to ExecuteScript, not a contained element you are trying to scroll in or out. If an element is inside custom scrollable container then yes, a custom scrollable container has to implement IScrollProvider.

We will look into fixing scrolling on Windows 10, but my first ideas for quick fix did not work, so it might take time. If you have resources for implementing relevant patterns for you custom elements then it might be a good idea as it will improve accessibility of your app and probably make testing easier.

[padmaja2812]

Thanks for suggestion. I created a scorllvierwerautomationpeer where i have implemented IScrollProvider. But now i m getting "Error reading JArray from JsonReader. Current JsonReader item is not an array: StartObject. Path", line 1, position 1." Request from outer server { "script": "automation: ScrollPattern.Scroll", "args": [ { "ELEMENT": "31160197-1" }, "{'v': 'smallIncrement', 'count': 10}" ], "SESSIONID": "e362ca2f-1f04-4624-ab28-06cc21d76acd" } I think error_ is because input format.Can you help me with correct input parameters?

[NickAb]

scroll_info should not be a string. It should be a dictionary. What language do you use to write tests? I assume you are writing tests in Java.

Map<String, Object> scrollInfo = new HashMap<String, Object>();
map.put("v", "smallIncrement");
map.put("count", 10);

driver.executeScript("automation: ScrollPattern.Scroll", element, scrollInfo);

[padmaja2812]

Yes i am using Java for writing test cases. I am still getting this error "Cannot access child value on Newtonsoft.Json.Linq.JValue." . Is it do with some version of Newtonsoft?

[NickAb]

Could you please provide driver logs that show request being sent, like you did in previous comment? Also, a code that you use to call executeScript will help.

[padmaja2812]

Hi Nick, It was a mistake from my side for last error. I have added scrollviewer to my custom contorl and i passing Id for that scrollviewer.

webelement wb=driver.appy.findElementByXName(name); Map<String, Object> scrollInfo = new HashMap<String, Object>(); scrollInfo.put("h", "smallIncrement"); scrollInfo.put("count", 10); ((RemoteWebDriver) driver.appy).executeScript( "automation: ScrollPattern.Scroll", webElement,scrollInfo); Is there any example which i can follow to implement these patterns?

Log 👍

19:17:14 [INFO] COMMAND findElement
{
  "using": "xname",
  "value": "FrmApp_scrollViewer",
  "SESSIONID": "5d786711-cd10-4fae-9ca4-bf112d413852"
}
19:17:14 [DEBUG] Sending request to inner driver: http://169.254.80.80:52524/
19:17:14 [DEBUG] RESPONSE:
OK:
{
  "sessionId": "5d786711-cd10-4fae-9ca4-bf112d413852",
  "status": 0,
  "value": {
    "ELEMENT": "25322943-2"
  }
}

19:17:14 [DEBUG] Client closed

19:17:14 [DEBUG] Waiting for a connection...
19:17:14 [DEBUG] ACCEPTED REQUEST POST /session/5d786711-cd10-4fae-9ca4-bf112d413852/execute HTTP/1.1
19:17:14 [INFO] COMMAND executeScript
{
  "script": "automation: ScrollPattern.Scroll",
  "args": [
    {
      "ELEMENT": "25322943-2"
    },
    {
      "h": "smallIncrement",
      "count": 10
    }
  ],
  "SESSIONID": "5d786711-cd10-4fae-9ca4-bf112d413852"
}
19:17:14 [DEBUG] Sending request to inner driver: http://169.254.80.80:52524/
19:17:24 [DEBUG] RESPONSE:
OK:
{
  "sessionId": "5d786711-cd10-4fae-9ca4-bf112d413852",
  "status": 17,
  "value": {
    "error": "javascript error",
    "stacktrace": "   at Winium.StoreApps.Driver.CommandExecutors.ExecuteScriptExecutor.ForwardCommand() in C:\\Users\\admin\\Desktop\\windows10\\Winium.StoreApps\\Winium\\Winium.StoreApps.Driver\\CommandExecutors\\ExecuteScriptExecutor.cs:line 59\r\n   at Winium.StoreApps.Driver.CommandExecutors.ExecuteScriptExecutor.DoImpl() in C:\\Users\\admin\\Desktop\\windows10\\Winium.StoreApps\\Winium\\Winium.StoreApps.Driver\\CommandExecutors\\ExecuteScriptExecutor.cs:line 91\r\n   at Winium.StoreApps.Driver.CommandExecutors.CommandExecutorBase.Do() in C:\\Users\\admin\\Desktop\\windows10\\Winium.StoreApps\\Winium\\Winium.StoreApps.Driver\\CommandExecutors\\CommandExecutorBase.cs:line 43",
    "message": "{\r\n  \"error\": \"javascript error\",\r\n  \"stacktrace\": \"   at Winium.StoreApps.InnerServer.Commands.CommandBase.InvokeSync(CoreDispatcher dispatcher, Action action)\\r\\n   at Winium.StoreApps.InnerServer.Commands.CommandBase.Do()\",\r\n  \"message\": \"Element is not horizontally scrollable.\"\r\n}"
  }
}

[NickAb]

Based on the video I assume you want to scroll vertically. Currently you are calling script to scroll horizontally. Please replace scrollInfo.put("h", "smallIncrement"); with scrollInfo.put("v", "smallIncrement"); or scrollInfo.put("vertical", "smallIncrement");. Sorry for misleading sample, I just copied it from the wiki without taking into account the direction of scroll.

Error "Element is not horizontally scrollable.\" should go away. If you will get "Element is not vertically scrollable.\", then there is a problem in you IScrollProvider implementation. Maybe Implementing the UI Automation Scroll Control Pattern will help you.

[padmaja2812]

I am getting this error "Element is not vertically scrollable.\". I have implemented Iscrollprovider and able to access it in my code AutomationPeer a = OnCreateAutomationPeer(); var s=a.GetPattern(PatternInterface.Scroll);

Whereas in winium code i m getting excecption after this line. var scrollProvider = element.GetProvider(PatternInterface.Scroll); And OnCreateAutomationPeer is not getting called automatically as it does in codedUI do I need to call after creation of every element ?

[NickAb]

Have you implemented VerticallyScrollable? It should return true if the element is vertically scrollable at the moment. You can reproduce minimal actions taken by Winium by running:

var peer = FrameworkElementAutomationPeer.FromElement(element);

if (peer == null)
{
    throw new Exception();
}

var s = peer.GetPattern(PatternInterface.Scroll) as IScrollProvider;

if (s == null)
{
    throw new Exception();
}

if (!s.VerticallyScrollable)
{
    throw new Exception();
}

s.Scroll(ScrollAmount.NoAmount, ScrollAmount.SmallIncrement)

Not that both VerticallyScrollable and Scroll are called and should be implemented.

[padmaja2812]

Hi Nick, I am able to scroll the custom widget problem was with creation of automation peer. All the testcases are same for phone8 and 10 so we need to modify many places in our current automation. It would be very helpful if the behaviour of scroll in both same,will be waiting for the fix. Thanks for your support.

PS:Can you suggest some links where in i can search and try to fix this issue.

[NickAb]

I was not able to find a quick fix before my travel. I am currently travelling and far away from work. I will be back on November 22 and will try to look at it this issue closer.

It looks like on Windows 10 hold action gets released and so instead of simulating swipe we get a bunch of clicks. You can see how gestures are handled here EmulatorController.cs. I think it is worth a try to look into migrating to IXdeAutomation API which includes some touch methods, but it will require some research, as there is no docs.