Make gamepads accessible by web worker

[tkodw]

I would like to see an extension to the spec that allows the gamepad API to be accessed directly from within web workers.

It is extremely difficult to avoid blocking and stalls in a window's main javascript context. Moving gamepad interactions to a worker avoids blocking related to other events flooding the window context and may reduce stalls related to garbage collection on some browsers.

With WebGL coming Web Workers, many more applications that make use of gamepads will likely be moving entirely within workers and will prefer to access gamepad data from workers instead of needing to read gamepad data outside of workers and send it there.

Gamepad access from workers is essential for haptic support. Haptic features that apply forces or limit motion require a high frequency feedback loop of reading controller inputs and updating force values. Outside of the trivial case of vibration, these features will not work well outside of workers.

[Mictronics]

Any updates on this topic?

[beidson]

Handling Gamepads from workers would be like handling mouse/keyboard/touch/pointer events from workers, which is (of course) not done.

Gamepad access from workers is essential for haptic support. Haptic features that apply forces or limit motion require a high frequency feedback loop of reading controller inputs and updating force values. Outside of the trivial case of vibration, these features will not work well outside of workers.

This is presented as a matter of fact without any supporting evidence.

[nondebug]

Hi Mictronics, I have a proposal for adding gamepad haptics and would appreciate your feedback:

https://github.com/w3c/gamepad/issues/19#issuecomment-321424335

Gamepad Haptics Extension

This proposal doesn't aim to support the type of haptics that would require a web worker (e.g. steering wheels and force feedback joysticks), but it would be good for us to understand where it falls short.

[Mictronics]

Handling Gamepads from workers would be like handling mouse/keyboard/touch/pointer events from workers, which is (of course) not done.

Unlike mouse/keyboard/touch or pointer inputs which are handled by events, the gamepad API requires polling to obtain input values. A polling job is definitely something I would like to handle in a worker and not burden the main thread with that. Especially in cases where the polling rate is not synchronized with the animation frame.

[GrosSacASac]

Maybe add a game pad button down event ( and more ) similar to how keydown works for keyboards. or maybe provide a builtin polling gamepad state function that is standard and high optimized by default

[marcoscaceres]

The use case will probably be covered by the events in #152 ... but exposing these things in a worker wouldn't make sense.

[vveisard]

This should be re-opened.

It's exactly as @Mictronics said, Gamepad polling is not events. The raw Gamepad object is functionally useless; you have to "sample" (reduce) multiple Gamepad polls into something usable (eg, to read a shoryuken input). If you have a high intensity game (like a fighting game) which needs a sample every game tick, and reduces multiple polls per sample, then you need to put your gamepad polling and sampling on another thread (preferably with the game).

[reillyeon]

Re-opening this issue because I agree that regardless of whether or not events are added to the Gamepad API sites should be able to receive and handle these events from a worker. Unlike mouse and keyboard input, gamepad input isn't tied to a particular DOM element and so doesn't have any of those issues in being brought to a worker.

@marcoscaceres, can you elaborate on your earlier message about exposing events in a worker not making sense?

[marcoscaceres]

@marcoscaceres, can you elaborate on your earlier message about exposing events in a worker not making sense?

Sorry, let me try again. We've been discussing adding an eventing model in #152.

Although it's true that today this is poling API, the proposal in #152 could change that.

I understand (and agree) that for a poling API exposing this in a worker makes a lot of sense for the reasons outlined by various folks above.

But if we change this to an event-driven API, does it make sense to expose it in a worker?

[reillyeon]

I agree that without the need for developers to do polling the motivation for moving that work to a separate thread is weaker.

It still seems beneficial however if developers are moving their game logic into workers anyways. This was the motivation of the original reporter of this issue.

[vveisard]

Is it possible to make the polling available in the worker threads, and keep event API only available in the main thread? (Unless the event API is meant to replace the polling...?)

[marcoscaceres]

(Unless the event API is meant to replace the polling...?)

It would make polling less necessary in a worker, but not replace it on the main thread (at least, we can't remove polling without "breaking the web"™️). However, as @reillyeon does reminds us, @tkodw is making the case that applications "will prefer to access gamepad data from workers instead" (https://github.com/w3c/gamepad/issues/37#issue-177528094).

So, some options to consider:

1. Add the eventing model, but only on main thread: not good, because canvas has been transferred to worker 👎 (and so has the logic).
2. Don't add eventing model, but expose Gamepad API (polling) in Worker: This is ok, but we need to make sure user activation still works (it should, in theory).
3. Expose Gampad API in worker, but only expose add eventing model. Question: is the polling API sufficient for all the gaming use cases?
4. Expose Gampad API in worker and the eventing model: is it worth it, from an implementation perspective, to add the events if all the use cases are by covered polling? Or is the eventing model of such advantage to developers that it outweighs the standardization cost?

Would love to hear thoughts/pros and cons!

[marcoscaceres]

As a counter to what I wrote above, I'd like to draw attention to what @beidson said (https://github.com/w3c/gamepad/issues/37#issuecomment-345017683):

Handling Gamepads from workers would be like handling mouse/keyboard/touch/pointer events from workers, which is (of course) not done.

Put differently, if you had a game that supported both Gamepad and keyboard input (and/or mouse), you would still need to keep a lot of the logic on main thread. I imagine most games fall under this category, supporting multiple input modalities for control.

So again, does it really make sense to privilege Gamepad in a worker when all other input devices only fire events on the main thread? Is a gamepad input so special as an input source (compared to mouse/keyboard/touch/pointer) that it warrants such special treatment in a worker scope?

On reflection, that claim now seems dubious to me. However, making Gampads behave like other input sources by using an eventing model does seem to make sense.

My feeling is that we should prioritize the eventing model - and if that proves insufficient, the come back and talk about Workers.

[7ombie]

Exposing the Gamepad API to workers is a basic requirement for most modern games.

The main thread is often too busy to respond to user input immediately, causing a noticeable delay. Games that require accurate input (rhythm games, fighting games etc) and games that already struggle with input latencies (anything with online multiplayer) cannot possibly tolerate the input latencies that are inevitable on the main thread.

The WebMIDI API is being updated to support MIDI in workers and audio worklets, specifically because realtime audio is not feasible with main thread latencies.

The only reason keyboard and pointer events are tied to the main thread is because they're part of the DOM API.

There is a proposal to expose keyboard and pointer events in workers. It hasn't made much progress, but threaded keyboard and pointer APIs will eventually happen, as many applications require them.

In any case, taking gamepad input, which is intrinsically global (unrelated to the DOM) and marrying it to the DOM's event system is a terrible idea, if that's what you're proposing to do. We already have a bunch of Web APIs that handle user input (WebUSB, WebMIDI, WebBlueTooth and WebSerial), which all have events, but none are tied to the DOM, and they're all available in workers (or will be eventually).

Conflating gamepad events with DOM events is a categorical error.

[7ombie]

I don't mean to rant, but the Web has an Input Problem. It is basically impossible to reliably respond to user input, of any kind, with low latency, on the Web. This has all kinds of subtle (but fundamental) implications that prevent the Web moving forwards. Low latency input is critical for a wide range of applications.

In every practical sense, the DOM API (with its DOM-based keyboard and pointer events) is inextricably tied to the main thread. Every other input API (gamepad, MIDI, USB, BlueTooth, Serial) should be available in workers (and worklets as appropriate).

Longterm, I hope (and expect) that we will also get some kind of new keyboard and pointer APIs that handle their events globally, and are available in workers. Regardless, the other input APIs (that are intrinsically global) should never be tied to the DOM or the main thread.

Note: None of this prevents an input API from being event-based or using promises etc.

The Gamepad API is currently contributing to the Input Problem. Please support workers.

[marcoscaceres]

This is valuable input (no pun intended) @7ombie. Personally I agree that we need to overhaul input events across the platform. As you said, it’s a larger project.

[7ombie]

@marcoscaceres - You've slightly misunderstood me.

Keyboard and pointer input needs more thought, but the other input APIs are fine, except for the Gamepad API.

WebUSB, WebHID and WebSerial are all available in workers. WebMIDI will support workers soon, and WebBluetooth intend to support workers, but haven't finalized the details.

So, while we generally need a proposal for threaded keyboard and pointer events on the Web, the Gamepad API should support workers as soon as possible, as every other global input API does or plans to do already.

[marcoscaceres]

WebUSB, WebHID and WebSerial are all available in workers. WebMIDI will support workers soon, and WebBluetooth intend to support workers, but haven't finalized the details.

Web MIDI aside, none of those are web standards unfortunately. It's hard to discuss the merits of those specs as they currently stand, as they are single implementer features that have, to date, received limited architectural input from multiple implementers. That's not to say that the model/architecture on which those specs are built is in any way deficient, but there has just been a lack of scrutiny so it's not possible to confidently say anything one way or the other.

as every other global input API does or plans to do already.

As above. There is no agreement in the web community that the model for the APIs you mentioned is correct and one the working groups should universally follow - so it would be premature to make any architectural decisions based on the specs you listed.

Having said that, it would be worth while having a generalized discussion with the TAG around this topic. In that context, WebUSB, WebHID and WebSerial, etc. would be great to discuss. It would also be great to also consider how the events model #152 could come into play here 🤔.

[7ombie]

@marcoscaceres - So long as nobody couples gamepads to the DOM (which would be crazy), this issue (gamepads in workers) is essentially API-agnostic. I'm not arguing for any particular model, API or architectural decisions, only that gamepads need to be available in workers.

The designers of every other relevant API accept the need for reliably handling user input with low latency, and all support workers or intend to. This general agreement is highly unlikely to change.

IIUC, the WebHID API does everything that the Gamepad API does (albeit at a lower level), without the limitations. If that's correct, there's a danger of the Gamepad API becoming redundant, as library authors implement more powerful gamepad APIs on top of WebHID.

EDIT: There are some non-HID gamepads, and there are a couple of other technicalities, so WebHID does not make the Gamepad API completely redundant.

[marcoscaceres]

@7ombie I’m open to discuss this further, but…

This general agreement is highly unlikely to change.

I’ve been chairing this working group for a while and this is honestly news to me? The Web Apps WG owns UI events, and a bunch of related specs.

It might just be that the discussions didn’t happen in the context of this W3C working group?

In any case, although it’s true that keyboard and mouse events can be bound to DOM elements, they can conceivably work without them… so what will we do about keyboard and mouse events if one is creating an application that works with multiple inputs? Won’t those events also need to be routed to a worker? Or is somehow gamepad events enough?

[7ombie]

@marcoscaceres - Ultimately, keyboard and pointer events will need to be exposed to workers (for example, you could image a DAW plugin that is controlled with a mouse, or a graphics app that lets you use touch gestures to draw and paint), but that requires new APIs for global keyboard and pointer events (unrelated to the DOM), as the DOM is pretty much married to the main thread.

The Gamepad API is not related to the DOM, so doesn't need a separate proposal to support workers.

Ultimately, if we want the Web to be capable of desktop-style applications, it needs to be able to handle every kind of user input in workers. Low latency response to user input is just that fundamental.

[marcoscaceres]

Thanks @7ombie, I just wanted to confirm. We coincidently do have some of the key people "in the room" to have this discussion (e.g., @nondebug, who edits Web HID and Gamepad, as well as @reillyeon, who has a ton of experience also in this area).

As was asked above, we still need some evidence to show that the current model is problem. This isn't to put it on you to supply it, but it's up to us to come up with something that shows the problem and then we can consider all the different options we have for addressing any shortcomings.

We also need to finish spec'ing out the eventing API (though it may not be consequential to this particular issue).

[Norren]

Having just stumbled across this in a late night hunt for the answers to this question along with other gamepad API questions for my own ElectronJS powered game, I would like to provide the long overdue context in long format, I see all this in the thread above but since @marcoscaceres wants evidence, I will restate it in the form of my real world problem. My apologies if being a little sleep deprived has made it into a bit of a ramble, but I feel the meandering explanation helps build the case, as it demonstrates a borderline layperson's approach to confronting the problem: (And perhaps if I have overlooked something obvious, the thing I have overlooked will be as or more informative to you as my plea itself.)

Presently, to render a WebGL powered 3D Engine to a Canvas, the most performant way to do this is to pass control of the Canvas element to a WebWorker via Offscreen Canvas. A RequestAnimationFrame() loop on the DOM drops animation frames inconsistently due to... blocking? (Yes, I think the term is Blocking) from a number of main DOM thread related background and foreground tasks while Offscreen Canvas has an entire thread to itself. Gamers always notice these frame drops when they happen as it usually negatively impacts their gaming experience, so we are forced to use Offscreen Canvas to not lose customers.

So thanks to Offscreen Canvas, the entire 3D Engine lives in a Web Worker along with the RequestAnimationFrame() loop that it uses to draw every frame, which naturally has to coexist with the actual game logic loop for basic important things like updating entity positions, checking for collision, etc. During this the game logic loop is expected to take input from the player... not play a potentially laggy game of telephone with the DOM 60 times a second in the hopes of eventually getting input from the player. For the same reason, input polling in the DOM is bad, because the DOM may defer JS events for some number of milliseconds while it resolves whatever blocking event is going on. Which gets us to the unreliable tool we have for handling that input polling.

Presently, the main thread has to take input from the player in an event (key/mouse) or redundant RequestAnimationFrame() loop (gamepad), and shoot that input over to the WebWorker as a message or write it to a SharedArrayBuffer. Neither is ideal as Messaging will arrive an unknown number of milliseconds later (necessitating rollback code for times when DOM blocking events cause the messages to arrive a few frames late) and SharedArrayBuffers apparently require something called Atomics to mitigate the race conditions caused when you have two separate threads pounding away at the same byte at 60 FPS or more, high refresh rate monitors allowing. Players notice when buttons "stick" or lag. Sometimes my Keyup events do not actually make it into the SharedArrayBuffer, and I have been putting off learning this new confusing Atomics thing as I am not a programmer, my background is in illustration and desktop IT support. I suspect I can get compensate for this without learning Atomics, but I find it ridiculous I have to jump through so many hoops, every one adds lag, and brings its own quirks, glitches, and unintended behavior.

Any reasonable developer using a compiled language would expect to be able to run some equivalent of if(keyboard.keys(KeyRotateRight).pressed || gamepad.buttons[ButtonRotateRight].pressed) tests, letting them have that data available on the very frame that the button is pressed, ready for the game loop, right there in the game loop where it is required.

In summary, being unable to test for input exactly at the point that input is required is inexcusable. The traditional Event based model works for interacting with DOM elements because adding a pointer event to the specific DOM element it controlled was exactly this, a test for input at the point the input is required. Now there is another location, the worker thread, that requires a test for input, and it should not have to play a game of telephone with another thread to get it!

You have your context for why this is needed now. Better 7 years later than never, but I do not want to be waiting another 7 years for a feature I needed to use earlier tonight. :)

[marcoscaceres]

Thanks @norren for sharing your experience. It certain adds weight to the argument and you are right that because of where the canvas is being rendered (off screen) and the close relationship between what is being rendered and user input, it does throw weight behind the argument that something is needed here. The situation is somewhat unique, in that we have forced developers into this situation precisely with postMessage() and SharedArrayBuffer, and I have to agree on a person level... that kinda sucks.

The argument about realtime access to inputs VS events is a little bit beside the point (the use cases can be met by either). I think browser vendors will tend to lean towards an eventing model (and an effort has been underway to make gamepad API more event based, with "live" objects #8). And even if one had "immediate" access to the state of some input device, as a developer you can't actually be guaranteed that it's not a cached value.

I can't promise this won't take another 7 years to fix, but we definitely need to have another serious conversation about this if postMessage and SharedArrayBuffer are causing noticeable issues.

[Norren]

Thanks for getting back to me, @marcoscaceres, this was an old enough thread I wasn't sure discussion would still be in play at all. With the benefit of having more sleep and the calmness that comes from having your code start working I can say that- although I have misgivings and dread about the how- I do have the SharedArrayBuffer working smoothly for the moment. (using temp vars to hold input on the DOM so that I only access the buffer once per index per animation frame, DOM-side) I am uncertain if that will hold once I get to implementing an audio worklet, but that is another day's problem. :)

As to polled inputs vs events, #8 is interesting. I will read through it and some others more thoroughly as spare time allows. From my perspective the two issues overlap heavily enough as to be the same, and I share the same misgivings I saw in a skim of in the thread about spammy events like "GamepadAxis" and "GamepadGyro". Spammy events. I'd imagine they will cause similar issues to what I was experiencing with dropped events while sending keyboard event data to SharedArrayBuffer, which in turn will make Worker access even more important.

I will direct any future commentary I have on the subject of events to that thread, however, to keep things on topic. Thanks again! :)

[7ombie]

@Norren - Thanks for taking the time to lay out your experience with the API.

To be fair, the issues you're having with postMessage and shared array buffers are intrinsic to threads on a multicore system. You must use atomics to access shared memory whenever you have one thread writing to memory that another thread may be reading from, whether you implement something yourself (using shared array buffers and atomics) or use something higher level, like postMessage (which can be slow, as it copies data, rather than sharing it).

The issue here is that your user input is only available on the main thread, while you need code in another thread to respond to user input with low latency. If you could respond to input in the worker thread directly, your issues with concurrency wouldn't apply here.

@marcoscaceres - The current (DOM-based) Keyboard API wouldn't make much sense in workers, as events would be global (there's no DOM). Given that and the fact that multiple threads may want to handle keyboard events, it seems to me that the API should allow threads (including the main thread) to register listeners for specific keybindings.

You'd have to bikeshed an API (looking at prior art), but the basic idea is to allow a thread to bind a listener to, for example, Ctrl+Enter, ignoring other input (rather than binding to every key-down event, then filtering out the ones you don't care about in the callback).

This would be a better API in many cases, so code running exclusively on the main thread may still use it. The current API would not be made redundant though, as it's part of the DOM API (permitting listeners to be bound to elements etc).

[7ombie]

Sorry, I forgot this was the Gamepad thread!

Regarding gamepads, I think they should work in the same way, in the sense that the API should be global (unrelated to the DOM) and available across all threads (including the main thread).

I don't see any advantage to 'splitting gamepads' (where for example, one thread handles button presses, while another handles analog sticks), but can imagine cases where different threads want to handle different pads (one pad per thread, for example).

So, the Gamepad API should basically allow a thread to take (and probably also later relinquish) ownership of a given pad, and thereafter, that pad's events would be exclusively handled (or polled for etc) by that thread.

[marcoscaceres]

Thank you for your valuable input, @7ombie!

It's been extremely beneficial to have this thread document the issues encountered, along with the interim solutions being used. This discussion helps us evaluate whether existing features, like atomics or shared array buffers, sufficiently address the concerns raised with the current API. This evaluation is crucial to determine if there's a need to expand the API's scope.

Furthermore, we appreciate alternative approaches to solving these challenges: They provide fresh perspectives on the problem space and potential solutions. That being said, as a Working Group Chair, I must remind everyone that discussing these matters in greater detail may require signing a contributor license agreement. This is to address potential intellectual property rights issues that could arise from non W3C member contributions. Please understand this isn't meant to discourage participation but to ensure legal protection for all involved.

For those outside the working group eager to contribute actively, we welcome your engagement through other channels. One such avenue we could explore is proposing something via the Web Incubator Community Group. Alternatively, we can bring in folks into the working group as "W3C Invited Experts" or have an existing member of the working group make a proposal that could address the issues above.

[7ombie]

I'm happy to sign the agreement, if it comes to that, but there's not much else to add here right now.

I would like to see some kind of initiative that looks into user input and workers very generally. Keyboards and pointers need to be exposed to workers, but they're beyond the scope of any of the APIs that are being discussed. There are also a handful of other input APIs that have communities, and have addressed workers to varying degrees, but could still do with some coordination, but again, there's no obvious place to have that conversation.

Thanks for taking the time to consider this, @marcoscaceres. Much appreciated.

[marcoscaceres]

I would like to see some kind of initiative that looks into user input and workers very generally.

@7ombie, ok, I'd suggest starting a conversation in on the working group mailing list: public-webapps@w3.org. The working group has all the right people, and it's within the scope of the working group.

I'd suggest just pointing out the problems being face by developers (as outlined above) and elude to the possible solution (i.e., "it would be beneficial to also have these events, or means to access the state of input devices, in web workers") but not propose an API or solution.

Would you be willing to do that? You could copy/paste come of the things already discussed above and make the case there.

[7ombie]

Would you be willing to do that?

@marcoscaceres - Yep. Will do. No problem. Gimme a day or two.

I'd suggest just pointing out the problems being face by developers (as outlined above) and elude to the possible solution [...] but not propose an API or solution.

Thanks for the advice. I'll keep that in mind. Thanks, @marcoscaceres. Much appreciated.

[7ombie]

Hey, @marcoscaceres. Hope you're well. I've been a bit busy, but have contacted the mailing list now (this evening). The title of the thread is Low Latency User Input.

EDIT: I had to verify my email address (I hadn't seen the response till now). I've done it. The message should show up fairly soon.

[marcoscaceres]

Thanks again for sending it, @7ombie. At least now we have something to point to.

For anyone following at home: https://lists.w3.org/Archives/Public/public-webapps/2023OctDec/0011.html

[marcoscaceres]

Proposed as a breakout session in March: https://github.com/w3c/breakouts-day-2024/issues/2

[marcoscaceres]

@Norren, why the confused response?

[Norren]

Just a little surprised at the lack of responses on the mailing list- wasn't expecting things to move fast, but the relative silence was puzzling.

Since the Breakout day session has been proposed, if that turns out to be a go, is that then open to the public?

[marcoscaceres]

Yeah, the mailing list is generally pretty quiet (see archive)... but it's still an important means of communication, even if people don't respond so much. Don't be discouraged thought, we will be able to get there right people together to discuss things... just that it might only be a small number of us who are interested in this topic, given how much stuff the web platform covers nowadays.

[7ombie]

Just glad to know that the issue has been raised and will be considered. Thanks again, @marcoscaceres.