Touch keyboard is not displayed properly on Windows

[mgnslndh]

Touching a TextBox does not bring up the keyboard. Touching "native" Windows text input like Notepad or similar will bring up the keyboard.

Expected behavior When I touch a textbox on a touch screen I want this keyboard to be displayed:

• Avalonia
  • Version 11.0.preview4
• Windows
  • Edition: Windows 10 Pro
  • Version: 21H2
  • OS Build: 19044.2364
  • Display Language: English
  • Keyboard Layout: Swedish / English

_Originally posted by @mgnslndh in https://github.com/AvaloniaUI/Avalonia/discussions/9974

[mgnslndh]

If it is possible it would be nice if Avalonia supported setting a property on the control that configures which layout of the touch keyboard will be displayed:

Numbers & Symbols:

Default:

The touch keyboard is styled differently on this comment compared to the image in issue description and I think it is because the image in the description might be from Windows 11 and the images in this comment is from Windows 10.

[maxkatz6]

@mgnslndh that's also something what we already support for mobile. I.e.:

<TextBox Watermark="Pin" PasswordChar="*" TextInputOptions.ContentType="Digits" />

Same API should be used for the windows.

[mgnslndh]

@maxkatz6 that sounds awesome. I have no idea what is broken but I'm willing to help out with testing or fixing the issue if someone can troubleshoot it to find the root cause.

[maxkatz6]

@mgnslndh tbh I am not an expert of low level IME integration, especially on windows. Maybe @Gillibald @yatli @Mikolaytis know better.

"imm32" is used for the IME support, and specifically ImmSetCandidateWindow is invoked to ask OS to show language specific IME window. It works with languages complex text input like Chinese and mouse input, but for some reason it doesn't show the keyboard on touch input for any other language. https://github.com/AvaloniaUI/Avalonia/blob/master/src/Windows/Avalonia.Win32/Input/Imm32InputMethod.cs#L244

[Sorien]

maybe this could help https://stackoverflow.com/a/40921638 and for changing layout setinputscope should be working https://learn.microsoft.com/en-us/windows/win32/api/inputscope/nf-inputscope-setinputscope when added here https://github.com/AvaloniaUI/Avalonia/blob/38fb9f18c463abb39eedd49e3f97004d6f7dc6f5/src/Windows/Avalonia.Win32/Input/Imm32InputMethod.cs#L268

[mgnslndh]

@maxkatz6 @Sorien Ok, so there are multiple ideas of how to control the touch keyboard on Windows. But there are some fundamentals that I would like to understand:

1. Is touch input for languages such as English considered complex text input and handled by Imm32? Or Should it belong in a separate implementation of ITextInputMethodImpl ?
2. Do we know if Imm32InputMethod works as intended for complex text input such as Chinese or is it broken, just like for ordinary touch with English language? Is it possible to test complex text input on Windows while still using English?
3. Is it the same Windows component (tabtip.exe) that will be displayed when Imm32InputMethod is used with complex language input such as Chinese, or will some other Windows component be displayed?
4. If the Windows component displayed is not TabTip.exe, which is used for touch input, how would we determine if one or the other should be displayed?

I guess the questions above are kind of flavors of the same one question :)

[tobyfirth]

Hi,

I have been doing a little bit of research on this. From my reading of Respond to the presence of the touch keyboard it seems to imply that the touch keyboard appearance is controlled by UI Automation. The AutomationPeer classes appear to be present in Avalonia so that might not be the case then.

For the input scope to show different keyboard layouts it looks like the interfaces from Text Services Framework would need to be implemented. Using the SetInputScope function is a possibility but the documentation suggests that applications should not be using this directly and implement the interfaces instead (ITfInputScope interface).

[mozesa]

Hello! I would like to inquiry if there is a chance to get working native OSK on Windows? Unfortunately, this fix was closed months ago and still no progress 😞

[timunie]

@mozesa if you know any, your contribution is welcome. Otherwise you will have to wait until we have one.

[mozesa]

@timunie I am starting to use this gist. Unfortunately, it is not the proper way for Avalonia.

Remark: I just copypasted from SO, links are there, I will definitely deal with it.

[tobyfirth]

@mozesa I have done something similar to you to get the touch keyboard to show for my project.

I found this library for WPF which has some helpful ideas for how to develop a solution.

I also found that if polling for the keyboard window sizes to check if already open/closed there were some small differences between windows 10 and 11 in what was reported.

[mozesa]

@tobyfirth I started also with that library, unfortunately that approach requires elevated rights (i.e. admin).

[mozesa]

I leave it here. There is eight-year bug description. The below code works, I mean, when there's touch capable monitor attached to the PC, and I click into the TextBox, the OSK shows up. But unfortunately, these methods are not reliably hiding - showing the OSK. I tried to specify not the Window but the Context but it would like to use UIContext. All in all, I am continuing...

using Windows.UI.ViewManagement;
using Adapter.Desktop.Views;
using Avalonia;
using Avalonia.Controls;
using Avalonia.Input;
using Avalonia.Interactivity;

namespace Adapter.Desktop.Pages;

public partial class ItemsPage : UserControl
{
    private IntPtr _handle;
    private InputPane _inputPane;

    public ItemsPage()
    {
        InitializeComponent();
    }

    protected override void OnAttachedToVisualTree(VisualTreeAttachmentEventArgs e)
    {
        base.OnAttachedToVisualTree(e);

        _handle = ((MainWindow)VisualRoot!).TryGetPlatformHandle()!.Handle;
        _inputPane = InputPaneInterop.GetForWindow(_handle);
    }

    private void InputElement_OnGotFocus(object? sender, GotFocusEventArgs e)
    {
        Console.WriteLine($"Show: {_inputPane.TryShow()}");
    }

    private void InputElement_OnLostFocus(object? sender, RoutedEventArgs e)
    {
        Console.WriteLine($"Hide: {_inputPane.TryHide()}");
    }
}

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <TargetFramework>net7.0-windows10.0.19041.0</TargetFramework>

[workgroupengineering]

see this

[tobyfirth]

@mozesa In terms of the library that I referenced it was the Observable sequence used in it that I found helpful. Actually showing/hiding the keyboard I use the ITipInvocation interface like you do in your gist.

I could never get IFrameworkInputPane interface to work for me.

I used AddClassHandler on GotFocusEvent, PointerPressedEvent, LostFocusEvent and KeyDownEvent to get the events which I fed into an observable sequence.

[Mr4g]

``Hello, if i use TextInputOptions.ContentType="Digits" my numeric virtual keyboard doesn't open.

            ```<Button Grid.Column="6" >
                <Image Source="{SvgImage /Assets/icons/gears.svg}"/>
                <Button.Flyout>
                    <Flyout>
                        <StackPanel>
                            <TextBox
                                Name="PasswordTextBox"
                                TextInputOptions.ContentType="Digits"
                                Text="{Binding EnteredPassword}"
                                Watermark="Wprowadź hasło" PasswordChar="*">
                                <TextBox.KeyBindings>
                                    <KeyBinding
                                        Gesture="Enter"
                                        Command="{Binding PasswordTextBoxPressedCommand}"/>
                                </TextBox.KeyBindings>
                            </TextBox>
                            <Button
                                Classes="TitleBarFlyoutsButton"
                                Name ="OpenFilesButton"
                                Content="OTWÓRZ PLIKI"
                                IsVisible="{Binding IsPasswordProtected}"
                                Command="{Binding OpenFilesButtonPressedCommand }">
                            </Button>
                            <Button
                                Classes="TitleBarFlyoutsButton"
                                Name ="ExitButton"
                                Content="EXIT"
                                IsVisible="{Binding IsPasswordProtected}"
                                Command="{Binding ExitPressedCommand }">
                            </Button>
                            <Button 
                                Classes="TitleBarFlyoutsButton"
                                Name = "MinimizationButton"
                                Content="MINIMALIZAJCA"
                                IsVisible="{Binding IsPasswordProtected}"
                                Command="{Binding MinimizeApplicationCommand}">
                            </Button>
                        </StackPanel>
                    </Flyout>
                </Button.Flyout>```

[tobyfirth]

I had another look at this and I think I have found what part of the UI Automation the touch keyboard responds to. According to this Microsoft example repo https://github.com/microsoft/Windows-universal-samples/tree/93bdfb92b3da76f2e49c959807fc5643bf0940c9/Samples/TouchKeyboard

On the PC, you can request that the touch keyboard display for a custom control by implementing the TextPattern provider interface (ITextProvider) and the ValuePattern provider interface (IValueProvider). Not supported on Phone.

This line is not in the latest version of the repo though.

I also found this post https://blog.tombam.net/implementing-textbox-with-on-screen-touch-keyboard-part-2/ which implements the ITextProvider and IValueProvider interfaces in WinForms. I have tested the code from the post and it does trigger the touch keyboard when the interfaces are used and the keyboard is not triggered when they are not so it is definitely the presence of these interfaces that controls the triggering of the touch keyboard.

It looks like IValueProvider is already implemented for Avalonia which leaves the ITextProvider (and ITextRangeProvider which is uses) to be implemented. Both of those interfaces are already defined in the Interop directory for Win32.

[timunie]

@tobyfirth if you have an idea how to implement that, consider to add a draft-PR for feedback and testing.

[tobyfirth]

@timunie I would like to do that. Not quite sure when I will have the time though. Hopefully in the next few weeks.

[Hackmodford]

@tobyfirth Did you ever get time to implement this?

[tobyfirth]

@Hackmodford Unfortunately not yet. I think I might be able to start it this week or next though.

[Hackmodford]

@tobyfirth Is the idea just that you'd need to add the ITextProvider to something like this?

[tobyfirth]

@Hackmodford Yes. And also the ITextRangeProvider. That is a little trickier as it has to deal with selecting a range of text and there are quite a few of functions to write.

I have now got a basic implementation, basically a copy of https://blog.tombam.net/implementing-textbox-with-on-screen-touch-keyboard-part-2/. I haven't actually tested it with a touch screen as the computer that I can write the code on does not have one. It looks alright when inspecting it with Accessibility Insights for Windows, at least it doesn't cause and crashes or exceptions. Hopefully tomorrow I can test the sandbox on a touch screen.

The main issue is that the code does not properly support selecting text ranges which will need some more work.

[federicocodo]

@tobyfirth Can you share the code? I too would need the OSK for avalonia for a project, I also have a Windows touch screen so I can check if it works and maybe help with the code

[tobyfirth]

@federicocodo It currently doesn't work and I am not sure why. It is probably something to do with either some specifics of what ITextRangeProvider is expecting and I am not returning or how it is all linked up in avalonia. I will try to at least get the branch up on my account this weekend.

[federicocodo]

@tobyfirth Thank you, that would be great

[tobyfirth]

I have push my not working version to my github account. Maybe someone else might be able to see something that I am missing.

Another possibility is that there is something that https://blog.tombam.net/implementing-textbox-with-on-screen-touch-keyboard-part-2/ example has that has been provided by default by winforms that the current avalonia implementation is missing.

[Laurnz]

I sometimes test touchscreen features on non touchscreen Windows devices like here. You have to install Visual Studio 2019, it does not ship with 2022. Also, do not forget to configure the touchscreen to show even when a keyboard is attached in the Windows settings.

@tobyfirth I have tested the code from the post and it does trigger the touch keyboard when the interfaces are used and the keyboard is not triggered when they are not

For me, the touch keyboard gets activated whether the interfaces are there or not in https://blog.tombam.net/implementing-textbox-with-on-screen-touch-keyboard-part-2/. Maybe I am doing something wrong, because the automation provider never even got instantiated for the example or the Avalonia implementation.

[tobyfirth]

That is odd. I will have a look at that program.

[Laurnz]

It seems like Microsoft has the Text Services Framework that it is being used for all on-screen keyboard related tasks. It apparently handles showing/hiding the OSK internally. The TSF is shortly explained by the MSEdge team here.

Here they are using it for WPF.

Maybe it would be worthwhile to also implement this for Avalonia.

[tobyfirth]

Using the windows simulator I didn't get the touch keyboard appearing when using my test app for avalonia but it did for the winforms app although I didn't try it with the added interfaces disabled.

Text Services Framework will be needed for things like showing the different keyboard layouts. When I originally looked at it seemed like it would be a lot more work to implement than adding ITextProvider and ITextRangeProvider.

And it doesn't look like Text Services Framework is needed as the core interface for that ITextStoreACP is not implemented in the .Net version of WinForms so was unlikely to have been in the framework version but the touch screen keyboard can still be made to appear by adding the UI Automation interfaces.

[maxkatz6]

@Laurnz yes, implementing TSF is definitely something we want to have at some point. But it's rather a substantial task, and not planned for next releases.

[Arthur-Camatti]

@tobyfirth Have you found any luck in making your version work?

Also, I'm too able to use the OSK on the example from Tombam's blog without the interfaces, even after replacing the custom TextBox with the default one. Seems like WinForms got an update on this since this solution.

[tobyfirth]

@Arthur-Camatti, I have also now found that the interfaces do not actually do anything anymore. Maybe at some point in the past. TSF looks like the correct way to get windows to recognise a text input and to trigger the touch keyboard if appropriate.

However, Avalonia does not currently support TSF so another way is needed. Since the release of 11.1 with the IInputPane interface the way to so this is a lot simpler. This gist is the current solution that I have created. I have only tested it as a proof of concept locally so there are probably edge cases that I have not encountered yet. It is also limited by the fact that I only want to use it for a fullscreen app so the code for moving the UI out of the way assumes that the window is fullscreen. If using with just a maximised window the the code for shifting the window is probably not what you want and you will need to come up with an appropriate alternative.

I hope that the code is still useful for you with these limitations.