POC for rudimentary touch event support

[jotaen4tinypilot]

Related https://github.com/tiny-pilot/tinypilot/issues/270.

This PR demonstrates an approach for how we could introduce/improve support for touch events. There is some complexity to this task, however, so I wanted to discuss my findings and how to proceed.

Notes on the code

• Considering the complexity and trickiness of the mouse-related logic in the <remote-screen> component, it would feel most robust and safe to me to isolate the touch-related code as much as possible. One way of doing that would be to transform the touch events to mouse events, and then proceed to act as if the touch input was actually mouse input. This seems to work okay here, though I haven’t tested more complex scenarios (more on that below).
• I’ve tested this on an iPad, and could see that this PR solves the following issues that we currently see there:
  • When clicking on the remote screen for the first time, the iPad pulls up the keyboard (not our onscreen keyboard, but the native iOS keyboard). I assume this is due to the <input> element. This undesired behaviour is prevented by calling preventDefault() on the touch event.
  • When tapping and holding, iOS opens the context menu of the browser. This is also prevented by calling preventDefault().
  • Tapping somewhere on the remote screen would position the target machine’s mouse cursor to that place, but it wouldn’t issue a click.

Mouse / touch actions

With this PR, the TinyPilot web UI would become generally usable on a touch device (as in: it’s not immediately and obviously broken). However: the only thing that currently works somewhat reliably with this PR is issuing a single left click if you touch somewhere. There are quite a few things left that we’d have to figure out:

• Right click / context click: this concept doesn’t exist with touch gestures. I think iOS devices usually emulate this if you tap and hold for like a second. On macOS, there is also a trackpad gesture where you tap with two fingers to open the context menu. I think we’d have to decide in which way we want to handle this, and implement that manually. An alternative could be to provide a separate button / menu item that issues a context click at the current cursor position. That’s not quite convenient, but would probably be relatively simple to build.
• Left double click: this only seems to work in theory. When I tested, I wasn’t able to tap twice in the exact same position, but there was always a slight mismatch of positions between both taps. With a macOS target machine connected, the target machine wouldn’t therefore recognise both taps/clicks as double click, due to the move in between. I’m not sure every operating system behaves like that, though. It might be possible to fix that by having some sort of timing logic that’s like: “if we see two touch events in rapid succession close together, we use the first position for the second tap.” Alternatively, we could also do a separate button / menu item here as well.
• Drag and drop: we’d have to listen and handle touchmove events for this to work. I haven’t looked into this, so I’m not sure how much work that would be, or whether there are any special cases that we’d have to account for.
• Hovering / moving the mouse without clicking or dragging: there is not standard way of emulating this, so if we’d wanted to support this, I think we’d have to roll our own approach.
• Wheel / scroll: I don’t think we can emulate this, unless we invent a custom gesture, or some other dedicated control.
• Touch gestures: for a robust implementation, we probably would have to detect if the user tries to perform a gesture, e.g. by using multiple fingers. We might want to disregard these inputs (unless we handle them in our own way), because it’s likely that the user tries to do something else – e.g., if they try to pinch with two fingers (in order to zoom), we shouldn’t interpret that as click or drag&drop action.

Testing

Another note is the hurdle of testing touch events. Browser dev tools have some basic support for emulating touch input, but that’s quite limited. At least in Chrome, I don’t think it’s possible to emulate multi-finger gestures, and there are also other quirks on mobile operating systems which we have to account for, for example the native keyboard potentially showing up, or the mobile OS or browser app intercepting certain gestures and trying to do its own thing.

As far as I see, the only reliable way of testing touch support is on an actual touch device. For the test to be meaningful, this probably has to be a tablet, not a smartphone, due to the screen size.

Discussion

I’m basically wondering how far we want to take this, and how much effort we want to invest:

• Which kinds of mouse actions do we want to support?
• How far do we want to optimise the touch behaviour? (Like on a scale from “completely broken” to “polished mobile-like native experience”)
• Which combination of mobile OS’es and browsers do we want to support or test on?

[jotaen4tinypilot]

I thought it would be worthwhile to preserve this proof-of-concept as is, so I’ve created a separate branch/PR for the first iteration → https://github.com/tiny-pilot/tinypilot/pull/1783.