Add OS image dragging to non-normative reasons for pointercancel

[appsforartists]

The nonnormative reasons for pointercancel include a bunch of scenarios where a browser default action (like panning) took over in the middle of a gesture. A common, but unmentioned, scenario is browser/OS-level drag-and-drop. Like when panning, I believe this should cause a pointercancel to be dispatched when the browser stops sending PointerEvents. Unfortunately, this doesn't seem to currently be specified; and none of {Chrome, Edge, Firefox} behave that way.

Here's a bug to fix in Chrome: https://crbug.com/720201

and a reduction: https://codepen.io/appsforartists/pen/NjybQX?editors=1011

[NekR]

What if one wants to track mouse position during dnd and highlight some things?

On May 10, 2017 5:25 AM, "Brenton Simpson" notifications@github.com wrote:

The nonnormative reasons for pointercancel include a bunch of scenarios where a browser default action (like panning) took over in the middle of a gesture. A common, but unmentioned, scenario is browser/OS-level drag-and-drop. Like panning, I believe this should cause a pointercancel. Unfortunately, this doesn't seem to currently be specified; and none of {Chrome, Edge, Firefox} behave that way.

Here's a bug to fix in Chrome: https://crbug.com/720201

and a reduction: https://codepen.io/appsforartists/pen/NjybQX?editors=1011

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/w3c/pointerevents/issues/205, or mute the thread https://github.com/notifications/unsubscribe-auth/ABIlkavVi1tN-oKrDd-aqIVxT6RZLJ3Wks5r4SAsgaJpZM4NWGra .

[appsforartists]

That's what the drag event type is for.

Currently, you will stop getting pointermove as soon as drag begins (in every browser). The bug is that there's no notification that you're going to stop getting events - they just stop. For instance, if you wanted to know how many pointers were currently down, you may increment a counter on pointerdown and decrement it on pointerup or pointercancel. If drag starting firing, you wouldn't receive either PointerEvent, so your counter would be permanently off by 1 every time an OS drag happened.

One solution would be to add dragstart to the list of events that decrements the counter. Unfortunately, this puts the author in the position of having to know every possible event that could cancel a pointer stream in every user agent. The beautiful thing about pointercancel is that it tracks all that complexity for you - so long as the browser reliably fires pointercancel before it stops sending pointer events, it's really easy for an author to reason about when his code is handling a pointer stream vs. when the browser is.

[NavidZ]

This is also related to #194 . I tried having dragstart as an event that cause the capturing to go away. If we fire pointercancel right before the dragstartthen it would cover both this and the capturing scenario. Is this solution somewhat reasonable and address the concerns?

[NekR]

Currently, you will stop getting pointermove as soon as drag begins (in every browser). The bug is that there's no notification that you're going to stop getting events - they just stop. ...

I see, sounds like a bug indeed.

[appsforartists]

I have updated the reduction. It now includes a second image that calls event.preventDefault() on dragstart. When that happens, pointermoves and eventually pointerup continue to be dispatched correctly, as if dragstart never happened.

pointercancel should only happen if the browser is taking over the pointer events and stopping the stream. Therefore, I think pointercancel should wait until dragstart knows if it has been prevented.

[NavidZ]

I don't think you can quite rely on that. If I recall correctly you can even prevent default the drag event and that would stop the drag as well. Also Esc key can stop the drag. So the drag doesn't necessary stops when the mouse is released. Looking at the spec it is mentioned that after dragstart event there should be only drag events fired to the elements and not mouse/touch events. So maybe if we wait until after dragstart to decide whether to fire a pointercancel/lostpointercapture is not a very spec compliant solution.

[appsforartists]

Sounds like that may be a bug/oversight in their spec as well. If a browser behavior interrupts a pointer stream, pointercancel should be sent.

Perhaps the right solution is to fire pointercancel after dragend if the drag was never prevented or cancelled (e.g. with esc). That would ignore any drags that the author/user prevented, but still ensure pointercancel fires in cases where the browser ceased a pointer stream.

[NavidZ]

I personally don't think delaying pointercancel until after dragend is the right approach. It is the same as saying delaying pointercancel until the scrolling is finished. I don't know why it matters whether drag is canceled or not. The stream of input was interrupted for some amount of time during the drag and pointercancel should have been sent before right before that interruption. I noted that your use case for counting how many fingers are down is also flawed. You cannot rely on the pairing of the events at all. You may receive a pointerup without receiving a pointerdown and vice versa.

[appsforartists]

I can see the argument for either sending it after a non-prevented dragstart or dragend. I'd be OK with either.

[NavidZ]

If we were to be compatible with DnD spec we should not be sending any event after dragstart and before dragend. Although the pointerevents are new anyways and violating that may not break any website. Do you have any scenario that might fail if we always send a pointercancel or lostpointercapture (if necessary) right before dragstart?

[appsforartists]

The scenario I have in mind is draggable on-screen elements. It ought to be reasonably simple to build basic drag and drop with pointer{down, move, up, cancel}. Inside down, I can register a listener for move. I can remove that listener on up or cancel.

If the Observables proposal lands, it would be effectively this:

// (presuming single pointer for brevity)
const drag$ = down$.flatMap(move$.takeUntil(upOrCancel$).scan(distanceFromLastEvent));

An author ought to be able to call dragStartEvent.preventDefault() to prevent that drag stream from being interrupted by a pointercancel. Otherwise, if the browser has taken control of the pointer stream, it should dispatch pointercancel.

[NekR]

@NavidZ

I noted that your use case for counting how many fingers are down is also flawed. You cannot rely on the pairing of the events at all. You may receive a pointerup without receiving a pointerdown and vice versa.

That's weird. Why isn't that guaranteed? How someone should be able to implement multi-touch events if pointerup may not be called? I thought that the purpose of pointercancel -- if pointerup cannot be sent because finger/mouse/whatever is interrupted or out of surface then pointercancel is guaranteed to be called. Also many scripts indeed register pointermove on pointerdown and unregister on pointerup/pointercancel. This also brings question about lostpointercapture which is supposed to be called on pointerup/pointerdown (i.e. implicit capture loose).

IMO, if this is true when that's a bug in either browser or in the spec. And that also makes Touch Events more attractable since they always report touches count.

[appsforartists]

@NekR I think his point was that if I pointerdown outside an element (which doesn't capture), and then pointerup on the element with the counter, the counter will be off.

[NavidZ]

@NekR

That's weird. Why isn't that guaranteed? How someone should be able to implement multi-touch events if pointerup may not be called? I thought that the purpose of pointercancel -- if pointerup cannot be sent because finger/mouse/whatever is interrupted or out of surface then pointercancel is guaranteed to be called. Also many scripts indeed register pointermove on pointerdown and unregister on pointerup/pointercancel. This also brings question about lostpointercapture which is supposed to be called on pointerup/pointerdown (i.e. implicit capture loose).

So the discussions on this issue are not only for touch. You can drag and drop mostly with mouse. Chrome does trigger drag action with long press touch as well. I don't know how Edge or other browsers do it. Regarding the matching there are quite a few scenarios that the assumption of matching pointerdown with up or cancel is wrong. Say you are in an iframe and somebody mousedown on your iframe and moves out. You will get a leave event but never any up or cancel event. The same happens if they press down outside and release inside your iframe. You will only get an up event. The scenario happens for touch as well if the capturing of touch is released. There are other scenarios like opening a context menu and stuff like that but I guess we are going out of context of this issue. In the nutshell, there are no guarantees to have a pointerdown matched with up/cancel in all the situations. I suggest filing another issue to talk about this matching if you are more interested.

@appsforartists If you want the disable browser drag and drop action shouldn't we use the declarative attributes like draggable=false to disable that behavior? Similar to touch-action:none that disables browser's scrolling action.

[appsforartists]

Looks like draggable=false only works on the element itself; whereas, dragstart bubbles.

If I'm listening to pointer events on a container, I ought to be able to reject drag on any of its children.

[NavidZ]

'dragevent' should be kind of the default action of pointerdown. So if you preventDefault pointerdown that would prevent any drag from happening in the future as the result of that pointerdown. Doesn't it?

[NekR]

Works for mousedown at least.

[appsforartists]

Hmmm… I was afraid downEvent.preventDefault() would suppress pointermove, but it doesn't seem to.

Still, when pointerdown fires, I don't know enough about the user intent yet to know if I should cancel. By the time I do, it's too late. Calling downEvent.preventDefault() after I have measured that the cursor has moved far enough to know I should be dragging doesn't stop click or scroll from firing.

What if there is a clickable element, like a <button>, in the draggable container? I would want the user to be able to click on it if they don't move their cursor but to drag the container if they do. Similarly, imagine a carousel that pans left/right. If the user drags left/right, my JS event listener ought to drag the carousel to show the next image. If the user drags up/down, the page ought to scroll (the browser default). In pointerdown, I don't have enough information to know if I should prevent the default behavior.

[mustaqahmed]

One key advantage of PointerEvents is that default actions are never blocked on script execution, to guarantee responsive browser behavior even with a slow script. This essentially means a script has less granular control over browser actions (vs TouchEvents): either the script declares beforehand (though touch-action) that it wants to handle the input event stream itself, or it has to take over the control at pointerdown. After the pointerdown, browser can take over the control anytime (but must let the script know through a pointercancel).

The drag action as the default action of pointerdown fits the model perfectly. I have no strong opinion on whether the pointercancel should be before or after the dragstart; I am slightly biased towards "before".

What if there is a clickable element, like a , in the draggable container? I would want the user to be able to click on it if they don't move their cursor but to drag the container if they do.

Same concept here: a script cannot delay a browser action until pointermove, it's too late to guarantee responsiveness. E.g. consider what would happen if the script is super slow in pointerdown.

Similarly, imagine a carousel that pans left/right. If the user drags left/right, my JS event listener ought to drag the carousel to show the next image. If the user drags up/down, the page ought to scroll (the browser default). In pointerdown, I don't have enough information to know if I should prevent the default behavior.

In this case, the page can limit browser's control to vertical panning only, though touch-action: pan-y.

[appsforartists]

@mustaqahmed I'm not sure I'm following your points. It sounds like you disagree with my objections about using downEvent.preventDefault() to prevent dragstart from triggering pointercancel, but I'm not sure you understand my concerns.

It sounds to me like the @NavidZ's recommendation so far is:

• pointercancel will be dispatched immediately before dragstart.
• If the author doesn't like that, the author ought to downEvent.preventDefault() to prevent drag from occurring.

My objection is that this isn't granular enough. Most default browser actions (like panning and clicking) should still be permitted. Therefore, I think pointercancel ought to only be dispatched if dragStart.preventDefault() hasn't been called, which means it should come after dragstart.

Does that make sense? Are we all on the same page?

[NavidZ]

@appsforartists your point is certainly valid and we have this case in other situations as well. Basically there might be multiple behaviors as the default action for some events and you either can preventDefault to prevent them all or not to call preventDefault and keep them all. Regarding this particular behavior discussed in this issue, do you have a sample page that preventDefaulting on down would prevent a behavior on your page on any browser? For example I don't recall any browser stop firing click due to preventDefault on down. For panning, if you are referring to touch as far as I know there is no way any browser does both panning and dragging at the same time.

@RByers @dtapuska are you aware of any cases that sending an input device event after dragstart could cause any compat issues? Note that the drag spec says there shouldn't be any input device event after drag event. Although at the time there was no pointer events.

[appsforartists]

@NavidZ thanks for challenging me to make a reduction - it's hard to keep track of which events do what sometimes in these threads. 😃

Here's the test you requested:

https://codepen.io/appsforartists/full/xdzBqy

scrolling and clicking both appear to still work, even when downEvent.preventDefault() is called, so perhaps that is also an acceptable strategy for disabling native drag-n-drop.

[NavidZ]

So are there any scenarios anyone can imagine that the following solution would fail:

• pointercancel (and lostpointercapture if needed) will be dispatched immediately before dragstart.
• If the author doesn't like that, the author ought to downEvent.preventDefault() to prevent drag from occurring.

@patrickkettner @smaug---- are you guys also happy with this solution? I'll give it a few days before modifying my patch to address this as well.

[smaug----]

dispatching pointercancel before dragstart sounds reasonable.

In which case would lostpointercapture be dispatched? I'd assume drag operation doesn't start if we're capturing.

[NavidZ]

We can go either way. The initial discussion was that drag operation will take priority over pointercapture (similar to pointerlock) and will cancel capture. At the same time you can always disable DnD using other ways and keep the pointercapture.

[smaug----]

Hmm, dragging takes priority over pointerlock?

[NavidZ]

No. I meant both dragging and pointerlock take priority over pointercapture.

[appsforartists]

I didn't realize pointer capture taking priority over dragging was an option: I like that idea.

Can you think of cases where someone would want both to set pointer capture and to have dragstart take it away? My hunch is that if someone has explicitly assumed pointercapture, they probably don't want it stolen back by DnD.

[smaug----]

yeah, I would assume the same. If one uses pointer capture, why would anyone expect dnd to occur?

[NavidZ]

That makes it slightly inconsistent. Because pointercapture is just a mechanism to reroute the input. Now if you capture the input to a draggable element what should the browser do? From the page perspective the draggable element is getting all the events and just doesn't get dragged? Doesn't this sound inconsistent?

[appsforartists]

I hear what you're saying, but I'm still having a hard time thinking of a practical reason to use pointercapture but also expect DnD.

[NavidZ]

I don't have any scenario in mind for expecting both capture and DnD at the same time. Also I don't have any strong feeling either way regarding priority of capture over drag or vice versa. The only thing I have for having drag taking priority over capture is the fact that currently Edge does that. So we just followed what we did for pointerlock in issue #194 .

Instead, we can have:

• pointercancel will be dispatched immediately before dragstart.
• If there was a pointer capture in place drag operation never starts (i.e. dragstart is never dispatched)

What do you think @patrickkettner as you lgtmed the issue #194 having other way around.

[NavidZ]

@patrickkettner ping. Can you take a look at this issue?

[patrickkettner]

Sorry, having a hard time following the thread here. What are you asking for my opinion on?

[NavidZ]

I tried to some up the decision that the people were inclined to in my last comment. Basically we would like to have:

• pointercancel will be dispatched immediately before dragstart.
• If there was a pointer capture in place drag operation never starts (i.e. dragstart is never dispatched)

However, this is not what Edge does. I believe in Edge if drag starts there will be a lostpointercapture after a few drag events and there is no pointercancel either. So we were wondering if the solution above would be something reasonable as is it something you can change Edge to match.

[RByers]

pointercancel will be dispatched immediately before dragstart.

What about mouse event compatibility? Is mouseup delivered reliably at the end of a drag sequence today? If we pointercancel then presumably we won't send pointerup at the end (which could be confusing for existing code looking for pointerdown and pointerup pairs). But would we still send a mouseup?

If there was a pointer capture in place drag operation never starts (i.e. dragstart is never dispatched)

This conceptually makes sense to me but I wonder about the web compat implications. Would this apply to the implict capture that occurs via touch also? Edge (and Chrome on ChromeOS) supports HTML DnD for touch, so there's some potential compat implications there.

That doesn't mean we shouldn't change though, but it means we'll want to make some attempt to quantify the compat risk. I need to read this thread in more detail and think through the implications. Other than being confusing, how urgent/important is this? A developer can always listen for dragstart to know they won't receive move events, and presumably developers who care about such things have long needed to do this for mouse events, right?

[NavidZ]

What about mouse event compatibility? Is mouseup delivered reliably at the end of a drag sequence today? If we pointercancel then presumably we won't send pointerup at the end (which could be confusing for existing code looking for pointerdown and pointerup pairs). But would we still send a mouseup?

Currently in both Edge and Chrome when drag starts there will be no more mouse events including move during the drag and the up that cancels the drag. Although if you cancel the drag with Esc you would get a mouseup event.

Regarding the capturing use cases, even if we ignore the pointercancel, when capturing is active and drag starts Chrome is broken in terms of sending correct capturing events and Edge is a little inconsistent as mentioned in issue 194. The discussions on that thread is also related. So we still need to decide what drag does with the capturing in progress and which one takes priority. Let me know what you think.

[RByers]

Currently in both Edge and Chrome when drag starts there will be no more mouse events including move during the drag and the up that cancels the drag. Although if you cancel the drag with Esc you would get a mouseup event.

Ah, OK so then there shouldn't be a compat problem sending pointercancel - sounds good to me!

I don't have a strong opinion on the interaction between capture and DnD. It sounds like you, @smaug---- and @appsforartists are on the same page that capture should take priority over DnD, and @patrickkettner hasn't provided a reason for why the Edge behavior is the opposite so capture taking priority SGTM.

[NavidZ]

@dtapuska mentioned a scenario that I missed earlier. @smaug---- @appsforartists, the long press touch causes the drag to start and since we have an implicit capturing for touches then if we were to give priority to capture over drag then the drag can never possibly start with touch on the web pages that don't know about pointer events and don't explicitly release the capture on touches! I don't think this is acceptable as it is a breaking change for dragging of touches.

As opposed to what I said earlier:

We can go either way. The initial discussion ...

we can not go either way. So basically capture cannot take priority over capture because I the reason I mentioned.

Circling back to my first solution:

• pointercancel (and lostpointercapture if needed) will be dispatched immediately before dragstart.
• If the author doesn't like that, the author ought to downEvent.preventDefault() to prevent drag from occurring.

I believe we agreed on this solution as well before this "capture priority over DnD" came into the picture. So are we still good with this solution here?

[appsforartists]

@NavidZ do you have a demo? I don't think I've ever seen a long press drag interaction: only a long press context menu.

Moreover, this repo is explicitly for pointer events. I'm not sure that the existing behavior for touch events should necessarily block doing the best thing here: you can't call setPointerCapture in a browser they doesn't support pointer events.

[NavidZ]

Demo for what? For long press drag? You can do it with any draggable tag. Like this: http://jsbin.com/giqopi/edit?html,output Just long press on the text. The implicit capture of touch pointer events are part of the pointerevents pec. I'm sorry that I wasn't clear in my previous comment but whenever I said touch I meant pointerevents generated by touch.

[appsforartists]

I see.

Would it be too complex to have pointer capture trump dragstart unless the draggable attribute is set?

[NavidZ]

I believe that if we design a consistent and predictable behavior it makes more sense to the developers. "When to drag" is a browser decision and you can disable those behaviors with some other means (as discussed earlier). But if I just give one out for draggable attribute to the pointer capture seems inconsistent to me.

[RByers]

Yeah this is what I was wondering about when I said:

Would this apply to the implict capture that occurs via touch also? Edge (and Chrome on ChromeOS) supports HTML DnD for touch, so there's some potential compat implications there.

I agree we don't want the priority to be conditional. Edge already makes DnD take priority over capture, I think being consistent with that is the best choice. So your plan SGTM.

[appsforartists]

I definitely hear the arguments for simplicity and consistency with existing APIs. There's a counterargument for explicitness and obviousness: I don't think I've ever seen HTML5 drag-and-drop used in a touch UI. Gestural interaction is the natural modality of touch UI. Favoring drag-and-drop at the expense of pointer events makes the common case harder to get right in order to support a feature (touch-based drag-and-drop) that isn't being used in practice.

Since pointer events are a new feature, I think there's still room to define how they interoperate with the draggable attribute. If draggable was popular, I'd be more amenable to allowing drag to trump pointer capture. However, because I don't believe it is, this is my proposal:

1. Explicitly captured pointers (e.g. by element.setPointerCapture) don't participate in HTML drag-and-drop: pointer{down,move,up} dispatch as if dragging doesn't exist, and dragstart et. al. never happen.
2. Explicitly draggable elements dispatch pointercancel & lostpointercapture immediately before dragstart. Calling downEvent.preventDefault() overrides this by preventing dragstart from ever happening.

This leaves the questions of what to do with touches.

Question: Does dragstart et. al. ever dispatch on a touchscreen if the draggable attribute hasn't also been set?

• If not, fantastic. That means touch pointers already behave as I've proposed (with the exception of sending pointercancel - a bug we need to fix). By making explicit pointer capture also suppress dragstart, we've made the API more predictable and consistent. 🎉

• However, if there is a case where a developer could expect drag on a touch device without explicitly using draggable, then we have a potential compatibility issue. In that case, perhaps the best solution is to dispatch pointercancel before dragstart for touch interactions unless setPointerCapture has explicitly been called. (@NavidZ's proposal)

My proposal is a pragmatic one - I realize it might be a bit gross as a spec. However, it does the right thing by default, while still supporting the uncommon use of draggable in conjunction with touch.

---

I think we all agree that when the platform isn't predictable and consistent, we have a problem. Unfortunately, that's already the case. An author who only knows about pointer{down, move, and up} can build a gesture recognizer now that works great on touch screens and usually for mice too. However, if the user happens to use a mouse and begin the gesture on an image or a link, HTML drag-and-drop kicks in (even if the author has explicitly used pointer capture to declare that they want to handle the gesture). Hopefully, the author catches this in testing, but both because content can be dynamic and because not everyone has the time and resources to test on every device, we know that in practice it can go uncaught, yielding a frustrating user experience.

After the author realizes that dragstart is interrupting the pointer stream, they then also have to know that dragstart is prevented by calling downEvent.preventDefault() (not the more obvious dragstartEvent.preventDefault()). This is an intimidating decision, because now the author also has to know all the default behavior that might follow a down event and opt in to cancelling all of it.

I'm concerned that the status quo requires the author to master too many separate concepts before effectively using pointer events.

The point of Pointer Events is to enable authors to implement gestural input once and to have it work consistently across input modalities. dragstart is literally interrupting that goal. By prioritizing pointer capture over HTML draggable, we minimize the number of concepts a developer has to master to get gestures right and eliminate an unfortunate footgun.

[NavidZ]

I definitely hear the arguments for simplicity and consistency with existing APIs. There's a counterargument for explicitness and obviousness: I don't think I've ever seen HTML5 drag-and-drop used in a touch UI. Gestural interaction is the natural modality of touch UI. Favoring drag-and-drop at the expense of pointer events makes the common case harder to get right in order to support a feature (touch-based drag-and-drop) that isn't being used in practice.

Can you go over the simple use case once again so we can focus on that part of the problem rather than introducing edge cases?

Explicitly captured pointers (e.g. by element.setPointerCapture) don't participate in HTML drag-and-drop: pointer{down,move,up} dispatch as if dragging doesn't exist, and dragstart et. al. never happen.

You are now differentiating between explicit and implicit capture which we wanted to indeed avoid. Now just getting gotpointercapture might mean either implicit or explicit which makes the prediction of the behavior that much harder.

Explicitly draggable elements dispatch pointercancel & lostpointercapture immediately before dragstart. Calling downEvent.preventDefault() overrides this by preventing dragstart from ever happening.

This whole drag action and how it starts is not spec'ed yet as far as I know. So this is a browser behavior and vendor specific. So it is not necessary only in this case. I'm more in favor of disabling drag beforehand declaratively similar to touch-action for scrolling so that browser can decide whether it wants to take over the input or not and if it does consume the input stream it just sends the pointercancel in all the cases no exception.

[NavidZ]

To push the issue forward and also address the use cases that @appsforartists brought up without adding more complications and edge cases I sent this pull request. This should be addressing all the usecases you had in mind @appsforartists. Right?

[RByers]

Chrome 62 has just rolled out to stable and there's a (google internal) report of at least one site being broken by the pointercancel that occurs during a drag.

@NavidZ @appsforartists can you summarize why you believe pointercancel is better than just what Edge is doing with lostpointercapture and pointerleave in this scenario? It seems reasonable to me that DnD would behave like some other element/frame coming in on top and capturing the pointer away (i.e. basically the same event stream as if I drag outside the window - we don't send pointercancel in that case either).

We've missed the boat for keeping this change from getting into Chrome 62, so we've got a couple weeks to see what other reports we get of breakage. If there's none then perhaps either approach is fine. But if there's more, and there's not really compelling reason why cancellation is better than leaving then perhaps we should reconsider this change?

[NavidZ]

The solution you are saying which was only sending the lostpointercapture was the one we explored here at first #194 . But then @appsforartists mentioned here in this comment that pointercancel is reasonable to be there. I couldn't argue against that as it also does sound reasonable to me due to the stream of events being terminated and handled by the browser which is the definition of pointercancel and couldn't imagine a case that it would break the existing apps. Also pointercancel thing would automatically cover the lose of capture. @appsforartists do you have anything else do add?

That being said as you mentioned I'd like to also see what the impact of the change is. We had another issue that we also agreed on the desired behavior and I had to revert the fix after getting reports of some broken sites. I'll keep an eye on anything that comes up.

[patrickhlauke]

for logical consistency it would make sense to fire pointercancel when the browser takes over the gesture (as happens, for instance, when the browser scrolls and that scrolling was not suppressed via touch-action).