Add an option to compute pointerMove coordinates from element top-left

[juangj]

I was talking to somebody about this at TPAC -- I'm pretty sure it was @shs96c, and maybe @AutomatedTester was also there -- but we didn't talk about it during the meeting proper, so I don't think it's in the minutes, and we don't seem to have written it down anywhere else.

The basic proposal was to create a way, for pointerMove actions, to specify that the x and y coordinates should be computed relative to the top-left corner of the element rather than the in-view center point of the element. The main benefit is to provide an easier transition from the Selenium JSON wire protocol, where the existing /moveto command uses the top-left corner.

This was also mentioned in https://github.com/mozilla/geckodriver/issues/789#issuecomment-310139137 but I don't believe a spec issue was raised at that time. The benefit raised in that thread is that coordinates relative to the top-left are much more intuitive for, e.g., canvas elements.

(To be clear: I am not proposing that this be in the Level 1 spec, especially at this point in the process.)

[andreastt]

So just to be clear, you are talking about the case where the origin field is a web element, and in this case the x and y fields are added to the in-view centre point of that element.

I would say this behaviour is inconsistent with the behaviour when you don’t pass a web element origin, where x and y has the meaning you desire: from the top-left corner of the viewport.

I would argue that using the in-view centre point when x and y are given is the wrong default behaviour because there is a later condition that checks that the calculated values are positive. This effectively makes it impossible to pass in a negative X or Y coordinate for clicking the upper left rectangle of the element.

I propose we change the action to not add x and y to the in-view centre point but from the top-left corner of the element’s bounding rectangle.

[juangj]

Are you thinking of Step 7 of the "dispatch a pointerMove action" algorithm?

If x is less than 0 or greater than the width of the viewport in CSS pixels, then return error with error code move target out of bounds.

My reading is that x is computed as x element + x offset in step 6, where x element is the in-view center point of the target and x offset is the value of "x" from the JSON command. Either of these values could be negative, but the x >= 0 check is only enforced on the sum.

My assumption was that the motivation here was a desire for a command with offset coordinates of (0, 0) to have the same meaning as a command that omits the coordinates. (I don't have any evidence that that was indeed the motivation, though. Perhaps it's buried in meeting minutes somewhere.)

Unfortunately, I also think the ship has sailed on this for Level 1, and it would be impolite to make the backwards-incompatible change for Level 2.

[andreastt]

Perhaps @mjzffr has some input on the way forward here. Since I don’t know the actions API intimately I am struggling to make up my mind what an idiomatic API would look like.

[mjzffr]

At this point, I don't feel I have a full enough picture to comment on whether the spec needs to be changed, but I can fill in some missing info.

In a discussion yesterday @AutomatedTester recalled that the motivation for using in-view center point in Actions was to help with compatibility with the Element Click command, since ultimately that command will be implemented with Actions under the hood.s

there is a later condition that checks that the calculated values are positive

That's not quite the case. The x/y offsets provided by the command can be negative. The condition is on the final destination of the pointer: however you calculate the destination coordinates of the pointer, they have to be within the viewport, otherwise you get a "move target out of bounds error".

Also in a discussion yesterday, we raised some questions about the claim that using top-left as origin avoids cases of move-target-out-of-bounds errors compared to the in-view-center-point origin, but now I can't find a reference to this anywhere in github/bug comments. @andreastt Can you point me to the right place or state the question we had if you remember it more accurately?

As an aside, if we make the origin top-left, I think we still need to ensure that it's in-view. Is that what other driver implementations already do?

[mjzffr]

Also in a discussion yesterday, we raised some questions about the claim that using top-left as origin avoids cases of move-target-out-of-bounds errors compared to the in-view-center-point origin, but now I can't find a reference to this anywhere in github/bug comments.

Oh, I see, that must have been referring to https://github.com/mozilla/geckodriver/issues/1123 In that case, yeah, action.moveToElement(pic, 1500, 150) (move to pic with offset 1500 and 150) relative to top-left corner will target 1500, 150 which happens to be in the viewport, but relative to in-view center point it will target 2008,408 which is definitely beyond the viewport and leads to an error.

Another option we could consider is exposing a command like getInViewCenterPoint(el) so that clients can more easily calculate offset coordinates to implement alternate [convenience] methods like move-relative-to-top-left.

[mjzffr]

The following comment is also relevant to this discussion: https://bugzilla.mozilla.org/show_bug.cgi?id=1429338#c4

[raguravindran]

I feel this has been left hanging in lot of places where I see similar questions. I recently had this problem with geckodriver 19 and FF 57. We also upgraded to FF 65 and gecko being 24. So, gecko is still continuing to have W3C specs intact but not the same with chromedriver (or not?).

We resolved this situation by,

WebElement ele = findElement(By.xpath("//div[contains(@class, 'columnClass')]"));

// Action to move to the element top-left corner
Actions userAction = new Actions(getDriver());
int getTopLeftY = ((ele.getSize().getHeight()/2) - ele.getSize().getHeight());
int getTopLeftX =  (ele.getSize().getWidth()/2) - ele.getSize().getWidth();
userAction.moveToElement(ele, getTopLeftX, getTopLeftY).click().perform();

From this if you want to move to any point at the element. you may want to

int positionX = getTopLeftX + 100;
int positionY = getTopLeftY + 100;

Which will click at the point where you are anticipating.

[raguravindran]

There seems to be a problem with FF 65 though. Has the specification changed I still see this INFO When using the W3C Action commands, offsets are from the center of element but for FF65 moveToElement(ele, 0,0) goes 60+ pixel more from the centre. I am really trying hard to get this work!. Any help or insights would be appreciated!