Compatibility mouse transition events should reflect a single logical mouse pointer

[NavidZ]

@RByers @mustaqahmed

Looking at this example:

https://output.jsbin.com/bupaluf

Step 1. Move the mouse into the green box and leave it there. Step 2. Touch the blue box and release. Step 3. Now slightly move the mouse.

Here is the sequence in case it is hard to test it on Edge:

---- Step 1 ---- pointerover (id=1) on green mouseover on green pointerenter (id=1) on green mouseenter on green ---- Step 2 ---- mouseout on green pointerleave (id=134) on green mouseleave on green pointerover (id=134) on blue mouseover on blue pointerenter (id=134) on blue mouseenter on blue pointerout (id=134) on blue mouseout on blue pointerleave (id=134) on blue mouseleave on blue ---- Step 3 ---- mouseover on green pointerenter (id=1) on green mouseenter on green

This sequence of events seem incorrect to me. Particularly looking at pointerevent with id=1 you can see the inconsistent sequence of events for it (i.e. 2 enters for green). Also mouse events (that are actually fired by moving mouse device) do no fully match the pointerevents with id=1 (i.e. the last 3 events).

The problem I see here is not quite addressed in the spec either. So I was wondering if something is missing or I have misunderstood something.

If there is actually a problem here, I can imagine the following solutions (which the first one looks cleaner to me but I don't know the impact).

1. Do not try to match pointerevents with mouse events at all. So basically we left mouse in the green box and we touched blue box and fired compatibility mouse events for the touch event. Now if we move the mouse since the pointerevent with id=1 is still in the green box we shouldn't send any transition events (i.e. pointerenter/over) however since the mouse (i.e. the compatibility mouse which might include the actual mouse, touch, or pen) is in the blue box we should now send the mouse transition events to both blue box (mouseout/leave) and green box (mouseover/enter). This solution breaks the consistency of pointerevent with id=1 with mouse events but tells a consistent story looking only at pointer events or only looking at the mouse events.
2. The second solution is when we touch somewhere and we fire compatibility mouse event we really move the mouse to there. So basically not only send the pointerevents with id=#RandomNumber for the touch event itself but also send the pointerevents with id=1 for the mouse. This solution makes sure the consistency of pointerevent with id=1 and the mouse events but in the cases like pen or touch events sends double pointer events which might not be desirable.

I was wondering what exactly Edge does as I couldn't quite figure out from the sequence of events. It is definitely neither of what I said.

[RByers]

This is a really interesting question, thanks @navidz!

Basically I'd summarize this as:

1. We'd want pointer events to represent a consistent enter/leave sequence per pointer ID
2. For compatibility, mouse events should always present a self-consistent sequence
3. But multiple different pointer IDs all map to mouse events
4. And the spec implies there should be a 1:1 mapping from each pointer event to a compatibility mouse event

We can't have all 4 properties. In the case above Edge seems to violate 1, but that seems like a real shame since it means developers using the new multi-point aware API don't get a self-consistent event stream. I'd personally rather violate 4 and maybe even 2 if necessary in edge cases.

@teddink can you describe Edge's behavior here and what you consider by-design vs. a bug?

Note that this is related to issue #7 which we still need to address in the spec somehow.

[teddink]

Appears that we have a bug in Edge - thanks for the discussion.

The behavior should be #1 as stated above, which translates to:

---- Step 1 ---- pointerover (id=1) on green mouseover on green pointerenter (id=1) on green mouseenter on green ---- Step 2 ---- pointerover (id=134) on blue mouseover on blue pointerenter (id=134) on blue mouseenter on blue pointerout (id=134) on blue mouseout on blue pointerleave (id=134) on blue mouseleave on blue ---- Step 3 ---- mouseover on green

[NavidZ]

But here if one only looks at the mouse events they have some events missing too. How come while mouse was on green on step 1 then all of a sudden blue div gets the mouseover/enter without green getting any out/leave in step 2?

[NavidZ]

How about something like this event sequence:

---- Step 1 ---- pointerover (id=1) on green mouseover on green pointerenter (id=1) on green mouseenter on green ---- Step 2 ---- mouseout on green mouseleave on green pointerover (id=134) on blue mouseover on blue pointerenter (id=134) on blue mouseenter on blue pointerout (id=134) on blue mouseout on blue pointerleave (id=134) on blue mouseleave on blue ---- Step 3 ---- mouseover on green mouseenter on green

[patrickhlauke]

unless i'm misunderstanding, i'd say that it's important that pointer events are consistent in the sequence of events they dispatch, but that it would require quite a bit of extra work to somehow define that the compat mouse events also need to have a self contained mapping. fundamentally, you're trying to shoehorn the concept of mouse (a single pointer device) into a multi-pointer environment. it can certainly be done, but basically you want for every pointer to always "close out" the compat mouse event stream from the idealised single pointer and then "enter" the context of where the pointer just acted. a lot of effort/keeping track i'd say.

my view would be: if a developer is only looking at mouse events in a multi-touch/multi-input environment, events which are there primarily for compatibility, then...they can expect some rough sequence jumps.

[NavidZ]

I agree with you Patrick. But in this spec we did mention mouseover/enter/out/leave events explicitly to be fired after corresponding pointer events. If you agree firing these events may already cause some inconsistency in the mouse event stream and to make them consistent we need more effort how about not mentioning those events in this spec at all.

Looking at this section: https://www.w3.org/TR/pointerevents/#compatibility-mapping-with-mouse-events

the suggested solution has the problem we talked about in this issue. Although I can see it mentions that this way of sending compat mouse events are optional and agents can choose a way of sending compat mouse events that is has most compatibility but maybe not mentioning the solution can open future avenues to have a better compat mouse events definition in some other spec or something.

Also I was wondering if you are aware of any developers that depend on the suggested solution in the spec (i.e. the mouse/pointer events in that sequence) when they run on Edge.

[mustaqahmed]

While waiting for more informed comments on the impact on developers, here are my two cents.

So far we have seen two possible solutions here: Solution A Maintain strict 1:1 mapping between pointer & legacy mouse events, but break the enter/leave/over/out consistency within legacy mouse events (as in @teddink's event sequence above). Solution B Make legacy mouse events consistent among themselves even in a multipointer environment, but break the strict 1:1 mapping between pointer & legacy mouse events (as in @navidzolghadr's event sequence above).

IMHO Solution B is better because it breaks only a suggestion in PointerEvent spec regarding "cross-spec" event sequences. On the other hand, Solution A breaks a behavior in an old spec which we may have to support forever (?).

[RByers]

In practice I suspect either option will be fine for web compat. There are already other ways the mouse event sequence can be violated (eg. window covered temporarily or window moved, many browsers don't properly report mouse state transitions). I agree that solution B is cleaner, but it's more different from what Edge has already shipped. If Edge folks have a reason to prefer solution A that's OK with me. I agree that in any multi-modality scenario mouse events are always going to be a hack, so anything that is shown to be largely compatible with the existing web is fine (and arguably even preferable to changing behavior in a way that could change site behavior in a new way).

@teddink Can you please link to a public bug tracking this issue in Edge?

[jacobrossi]

@rbyers here's a link to the bug https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/6773357/

[RByers]

Thanks! Notes from discussion on the call today are here.

Our main concern is the potential for broken-looking UX when switching between mouse and touch input devices on a website designed only for mouse events (not at all an uncommon scenario). Eg. move the mouse over one hover menu then tap on another as in this example. With current Chrome and Edge behavior you shouldn't get into a state where both menus are open at once, but with Ted's proposal you would.

Also it seems wrong for the pointer events spec to define a sequence of mouse events which is inconsistent with the UI Events spec.

Perhaps we could simplify this by having the PE spec define the behavior of a virtual "legacy" mouse device, and leaving the specific generation of mouseover/enter/leave/out events up to the definition in UI Events? @mustaqahmed is going to propose something along these lines.

[RByers]

On the call we agreed that this high-level direction was good. We'll need to write some good web-platform-tests to cover this behavior change, and @teddink says the Edge team is looking into how hard it will be to implement the changes (so may need to come back with implementation-motivated tweaks). But we're comfortable moving in this direction.

[RByers]

Spec is updated, leave the issue open pending some new / updated tests?

[mustaqahmed]

Closing this since the spec issue is now resolved. I will create a new issue in w3c/web-platform-tests.

[mustaqahmed]

Added the following web-platform-tests issue for the missing test: Need tests for compat mouse events for multiple pointers.

[mustaqahmed]

@RByers, @teddink: ptal at the PR for a new test.

[patrickhlauke]

Just to check, this whole discussion is the reason why on Android, with a paired bluetooth mouse and Chrome 53 Dev with enabled PE, I get some "interesting" results, right? (or is this a separate weirdness, due to the fact that a mouse on Android also "fakes" touch events?)

https://patrickhlauke.github.io/touch/tests/results/#mobile-tablet-keyboard-mouse-events

moving mouse to my test button: pointerover > pointerenter > mouseover > mouseenter

clicking the mouse button (note the first 6 events, where the pointer is going out before coming back in, effectively): pointerout > pointerleave > mouseout > mouseleave > pointerover > pointerenter > pointerdown > touchstart > (gotpointercapture) > pointerup > pointerout > pointerleave > (lostpointercapture) > touchend > pointerover > pointerenter > mouseover > mouseenter > pointermove > mousemove > mousedown > focus > mouseup > click

second click (note the first 6 events, where the pointer is going out before coming back in, effectively): pointerout > pointerleave > mouseout > mouseleave > pointerover > pointerenter > pointerdown > touchstart > (gotpointercapture) > pointerup > pointerout > pointerleave > (lostpointercapture) > touchend > pointerover > pointerenter > mouseover > mouseenter > pointermove > mousemove > mousedown > mouseup > click

move mouse out of button: pointerout > pointerleave > mouseout > mouseleave > blur

[RByers]

I think it may be related, but it's not the only issue you're seeing. We also just have a couple (what we've now decided are definitely) bugs on Chrome for Android with how we handle mouse. Once we've got those bugs fixed, we should retest this scenario to make sure we handle it properly. There's no reason Android should do anything different than Chrome desktop for these touch+mouse scenarios.