AshleyScirra commented 3 months ago

Bug Report

The Gamepad API does not work in WKWebView. This affects any content published to iOS with Cordova that uses gamepad input (such as games, as used by Construct).

According to Apple the problem may be that WKWebView needs to be the firstResponder. I'm filing this issue as cordova-ios may need to configure the WKWebView accordingly.

Problem

Repro Xcode project made using Construct: Gamepad control example.ios.project.zip

Steps to reproduce:

Connect a gamepad to an iOS device (in my case I used bluetooth to pair the controller to the device)
Launch the Xcode project on the iOS device
Press some gamepad buttons

What is expected to happen?

The app should detect the gamepad upon first pressing a button and allow controlling the sprite with gamepad input. This is how it works in the Safari browser.

What does actually happen?

No gamepad input is ever detected.

Information

Originally I assumed this was a bug in WKWebView and filed it with Apple here: https://bugs.webkit.org/show_bug.cgi?id=269292

They responded:

To expose Gamepad support, your WKWebView needs to be the firstResponder.

Safari - correctly for Apple platforms - keeps focused WKWebViews as the firstResponder.

Many apps neglect to do this, and once we let a developer know and they make the change gamepads start working.

So it may be the case that cordova-ios needs to configure WKWebView as they recommend.

Command or Code

See repro above. It ought to reproduce with any example that uses the Gamepad API if you want a more minimal repro.

Environment, Platform, Device

Tested on iPhone 12 Pro Max with iOS 17.2.1 and a PlayStation 4 wireless controller.

Version information

Using cordova-ios@7.0.1

Checklist

[x] I searched for existing GitHub issues
[x] I updated all Cordova tooling to most recent version
[x] I included all the necessary information above

dpogue commented 3 months ago

hmmm, in theory we tried to fix this in https://github.com/apache/cordova-ios/pull/1337 which should be included in cordova-ios 7.0.1

dpogue commented 3 months ago

@AshleyScirra I tried to test this (on cordova-ios master branch) with a hardware keyboard attached to an iPad, and it seemed like the Cordova WKWebView was receiving events properly with no need to tap the screen. I don't have a gamepad to test the specific API, but as far as I can tell it is correctly setting the webview as the first responder after the splash screen vanishes.

Are you doing anything custom in terms of splashscreen or using CDVViewController directly?

AshleyScirra commented 2 months ago

I don't believe we're doing anything custom with the splashscreen or CDVViewController .

AshleyScirra commented 2 months ago

A corresponding Chrome for iOS (which uses WKWebView) issue has apparently been fixed now: https://issues.chromium.org/issues/325307469

A Chromium engineer commented:

The gamepad API doesn't work if the WKWebView is not first responder at "the correct" point during a page load. I didn't dig into WebKit to see exactly where this is happening, but you can test with this attached sample app.

Perhaps that's relevant to Cordova too?

michaeldo1 commented 2 months ago

Based on my testing in Chrome, I believe the change in #1337 happens too late to fix the Gamepad API. showLaunchScreen happens in onWebViewPageDidLoad after a delay of fadeSplashScreenDuration. In my testing, any delay more than "0 seconds" after requesting the webview to load would break the Gamepad API.

I attached a sample project to the WebKit bug in comment7 which can be used to test.

Note that keyboard commands appear to work even if the first responder is set later, but Gamepad API doesn't. (This seems like it could be a WebKit bug.)

@AshleyScirra If a change is made in this project, it needs to be debugged further with real hardware in order to find the right place to becomeFirstResponder without breaking any other Cordova functionality. Alternatively, digging into WebKit and suggesting a patch there to accept GamePad API hardware connections even if the firstResponder is set later on (similar to keyboard command handling) could also fix this.

dpogue commented 2 months ago

Ahh, that's probably the issue, since we aren't marking the webview as first responder until after the page has finished loading.

I wonder if there are any accessibility concerns with setting the first responder immediately on the webview, even while the splashscreen is displayed...

dpogue commented 1 month ago

I've confirmed that making the WKWebView the firstResponder immediately does make the gamepad API work as expected. I still don't love that solution so I might see if it can be fixed in WebKit, but we'll probably need this as a fallback for existing iOS versions.

AshleyScirra commented 1 month ago

Thanks - appreciate the work on this.