WebView2 focus trap on UWP XBOX when using XY directional navigation

LDS2020 commented 9 months ago

What happened?

While testing the WebView2 WinUI2 Sample on Xbox, along with an app I have in the MS Store, I find that I am unable to disengage from the WV2 control when using XY directional navigation. I am not sure if this is an implementation issue in the Sample or a bug in WebView2 for Xbox. I find it hard to believe this issue didn't get picked up in testing, so I am starting here.

The existing UWP sample works fine on Xbox but it uses Mouse or Pointer mode, which allows the users to move from the Webview to other controls on the page. An Xbox app, however, would normally use XY navigation which allows full access to every button on the Gamepad and is the #1 best practice for Xbox UI.
When you switch to XY navigation, the user gets stuck in a a focus trap on the WebView2 control and cannot disengage by pressing the B button which is how most controls (like the address bar in this sample or the original WebView1) work. The only way to exit is click the Xbox button, Quit the app and start again.

Importance Blocking. Although it is theoretically possible to switch Xbox apps over to Mouse mode, it would require major changes for developers and a foreign user experience for the users.

Environment

Runtime Channel Stable release (WebView2 Runtime)

Runtime Version Pre-installed on Xbox

WebView2 Version 1.0.2210.55

Framework WinUI2/UWP Microsoft.UI.Xaml 2.8.6

Operating System Xbox

OS Version 10.0.25398.2921

Repro steps

See this modified repo of the WebView2 WinUI2 sample app, where I have:
Switched to XY Navigation Mode by changing PointerMode to WhenRequested in App.xaml.cs
Turned on Reveal Focus and written debug lines, so you can see which control has focus
Disabled scaling to avoid the bug documented in WebView2 issue 4133 by @remokeitel
Set IsFocusEngagementEnabled=true, RequiresPointer=true and set IsFocusEngaged=true for the WebView2 control

Expected behavior

The user should be able to navigate to the WebView2 on the page, select A to engage and interact with it using the temporary Mouse pointer, and select B to disengage so they can select another control, as described in this quote from here:

When the IsFocusEngagementEnabled property is set to true, it marks the control as requiring focus engagement. This means that the user must press the A/Select button to "engage" the control and interact with it. When they are finished, they can press the B/Back button to disengage the control and navigate away from it.

Instead, the user is stuck on the WebView control, as shown in this screenshot, and cannot get to any of the other controls on the page or use any of the buttons on the gamepad:

WinUi2 WebView2 example

Related Issue A separate but related issue is that WebView2 for WinUi2 has not yet implemented Key events like OnPreviewKeyDown, as has been done on other platforms, so it is not possible to intercept the B button inside the Webview2 to code a workaround.

AB#48450583

jennifer-jacobs commented 9 months ago

Hi @LDS2020,

I can repro this as well and sorry you are hitting this. Most of our testing was done in scenarios where the app was entirely a WebView and not as much mixed XAML and WebView content. This has been added to our backlog!

jennifer-jacobs commented 9 months ago

In order to better investigate this, do you have a sample WV1 app that shows this behavior working?

Do you own the content that is being shown? If so, as a workaround you could intercept the B message using a key listener on the page and send a message back to XAML and manually set focus back to the correct element.

LDS2020 commented 9 months ago

@jennifer-jacobs, thanks for confirming that you can reproduce the error and I will follow this issue for future updates.

to see how it works with WV1, check out my app CleanOut for Xbox or, if you need source code, let me know if you want me to create a version of the sample app using WV1?
forgetting WV1 though, WV2 on Xbox should behave just like the TextBox AddressBar does on the Browser.xaml page. Press A to interact with it and B to escape, and you can tab over it without engaging if you want to.
as I stated in my Related Issues comment above, you can't code a workaround to intercept the B button because the key listeners have not been implemented in the WinUI2 version of WV2, like they have on WPF. You can wire up the event handlers at either the WV2 control or Page level but they don't do anything when the WV2 is FocusEngaged. The WV2 control intercepts everything. Let me know if I am wrong about that?

remokeitel commented 9 months ago

Hi @LDS2020,

you can use (as a workaround) the Gamepad API to get pressed buttons:

webView2.CoreWebView2.WebMessageReceived += (coreWebView2, args) =>
{
    var key = Enum.Parse<VirtualKey>(args.TryGetWebMessageAsString());
    switch (key)
    {
        case VirtualKey.GamepadB:
            //...
            break;
        //...
    } 
};

Add the following JavaScript code to the WebView2 HTML:

let interval;
window.addEventListener(
    "gamepadconnected",
    (e) => {
        interval = setInterval(() => {
            for (const gamepad of navigator.getGamepads()) {
                if (gamepad != null) {
                    if (gamepad.buttons[0].pressed)
                        window.chrome.webview.postMessage("GamepadA");
                    else if (gamepad.buttons[1].pressed)
                        window.chrome.webview.postMessage("GamepadB");
                    else if (gamepad.buttons[2].pressed)
                        window.chrome.webview.postMessage("GamepadX");
                    else if (gamepad.buttons[3].pressed)
                        window.chrome.webview.postMessage("GamepadY");
                    else if (gamepad.buttons[4].pressed)
                        window.chrome.webview.postMessage("GamepadLeftShoulder");
                    else if (gamepad.buttons[5].pressed)
                        window.chrome.webview.postMessage("GamepadRightShoulder");
                    else if (gamepad.buttons[6].pressed)
                        window.chrome.webview.postMessage("GamepadLeftTrigger");
                    else if (gamepad.buttons[7].pressed)
                        window.chrome.webview.postMessage("GamepadRightTrigger");
                    else if (gamepad.buttons[8].pressed)
                        window.chrome.webview.postMessage("GamepadView");
                    else if (gamepad.buttons[9].pressed)
                        window.chrome.webview.postMessage("GamepadMenu");
                    else if (gamepad.buttons[10].pressed)
                        window.chrome.webview.postMessage("GamepadLeftThumbstickButton");
                    else if (gamepad.buttons[11].pressed)
                        window.chrome.webview.postMessage("GamepadRightThumbstickButton");
                    else if (gamepad.buttons[12].pressed)
                        window.chrome.webview.postMessage("GamepadDPadUp");
                    else if (gamepad.buttons[13].pressed)
                        window.chrome.webview.postMessage("GamepadDPadDown");
                    else if (gamepad.buttons[14].pressed)
                        window.chrome.webview.postMessage("GamepadDPadLeft");
                    else if (gamepad.buttons[15].pressed)
                        window.chrome.webview.postMessage("GamepadDPadRight");
                    //else if (gamepad.buttons[16].pressed)
                    //    window.chrome.webview.postMessage("GoHome");
                }
            }
        }, 100);
    },
    false
);
window.addEventListener(
    "gamepaddisconnected",
    (e) => {
        if (navigator.getGamepads()[0] == null &&
            navigator.getGamepads()[1] == null &&
            navigator.getGamepads()[2] == null &&
            navigator.getGamepads()[3] == null)
            clearInterval(interval);
    },
    false
);

But I hope in the near future we will get functional key event listeners for the WebView2 control!

LDS2020 commented 9 months ago

Thanks @remokeitel! I will give that a try as a workaround.

@jennifer-jacobs I noticed Victor tagged this issue as Tracked, but not as Bug yet, even though you were able to repro the issue. Does this mean you have not determined if it is an implementation issue or a WV2 defect yet?

jennifer-jacobs commented 9 months ago

@LDS2020 We haven't determined the right fix for it yet (whether it's an implementation or WV2 bug). We've added it to our backlog to continue looking into it and I've marked it as a bug. Thanks for calling that out!

MicrosoftEdge / WebView2Feedback

WebView2 focus trap on UWP XBOX when using XY directional navigation #4284

What happened?

Environment