Support more numeral systems

[ivadham]

Problem Statement Calculator only supports English (Western) numeral. If I change my keyboard layout and enter (or paste) Persian numeral, it simply does nothing!

Proposal Support more numeral systems as input, but show them as English (Western) numeral. Examples: Hindu–Arabic numeral system 0 = ۰ 1 = ۱ 2 = ۲ 3 = ۳ 4 = ۴ 5 = ۵ 6 = ۶ 7 = ۷ 8 = ۸ 9 = ۹

Goals I can enter digits with any keyboard layout, pretty fast, no worries.

[rudyhuyn]

Does it work if you change the language of the OS (in windows settings> Languages)? Based on the source code it should, but I agree, keyboard layouts should also be managed.

[ivadham]

@rudyhuyn My Windows display language is English (United States), and I need to download Local Experience Packs to change it, but that's not the point. As you said, Calculator should be able to get numeral input, regardless of keyboard layout.

[miloush]

I don't think this has much to do with keyboard layouts. You just need to go through the chars in the pasted text and call something like char.GetNumericValue on them, hoping the number is in a decimal system.

[rudyhuyn]

The application uses the Region setting of your PC to selected the current digits to use (0-9), (۰ - ۹), ..., not the keyboard layout you use (I can use a AZERTY keyboard while the OS is set for English US). It will this setting to add shortcuts to buttons and to parse the Clipboard.

We can easily detect the keyboard layout used and fix the clipboard with a string.replace, but the real issue is with how the application manages the keyboard, the application doesn't observe keyboard events, instead, it associate each UI button to a keyboard key (not even code-side but directly in the .resw), so when you type '1' in fact, in fact it "presses" the UI button "1". It's currently not possible to associate more than 1 shortcut (see KeyboardShortcutManager) to a key, so if you want to allow users to type 9 and ۹, a refactoring will be necessary.

This is why I asked @ivadham to try to change its region, to confirm the issue was limited to keyboard shortcuts and not deeper

[miloush]

I was covering the paste scenario, but it can be the same for the direct input - when the received character does not map to any button, you can try to get the decimal digit and try again. That way you don't need to associate more than 1 shortcut and hence no refactoring would be necessary.

[ivadham]

OK, I did what @rudyhuyn asked me to do:

• Windows display language: فارسی (Persian)
• Regional format: Persian (Iran)

Now I can paste Persian numeral, but STILL can't enter/type Persian numeral. I copy-pasted ۲۲۲333۴۴۴ and it worked!

FOR COMPARISON, I DID THE SAME TEST WITH FOLLOWING SETTINGS:

• Windows display language: English (United States)
• Regional format: English (United States)

I copy-pasted ۲۲۲333۴۴۴; no luck...

[MicrosoftIssueBot]

This is your friendly Microsoft Issue Bot. I've seen this issue come in and have gone to tell a human about it.

[miloush]

My bad, this needs API that returns _NumericValue as per UAX #44 (i.e. equivalent to System.Globalization.CharUnicodeInfo.GetDecimalDigitValue), however, as far I can tell there is no such API in UWP, there is only Windows::Data::Text::UnicodeCharacters::GetNumericType which just returns _NumericType.

Refactoring to support multiple text buttons is going to be nightmare. There are currently about 50 decimal systems (12 languages have one ore more keyboard layouts without ASCII digits at all), and all of them should work regardless of the localization, since users can use the keyboard layout while having UI in English, as was the original use case in this issue.

So you can either implement the mapping yourself which would have to be maintained in the codebase, or interop which doesn't make sense. I believe the correct thing to do here would be to file a request for new UWP API, most likely UnicodeCharacters::GetNumericValue, and use it when available.

[rudyhuyn]

You can detect the keyboard layout via the class CoreTextServicesManager, so we need to manage only 2 locales: the one related to current Region and the one related to the keyboard layout. The 2 challenges are:

• The current architecture only supports 1 locale as mentioned before
• country to the Region, the keyboard layout can be changed while the app is running, we need to check the value on the fly.

[miloush]

I do not dispute that would be an improvement to the current behaviour. If that's a low hanging fruit, go for it.

If I understand it correctly:

• If a user has the US keyboard under Arabic input language and the UI/Region is in Arabic, both your sources would say Arabic and typing numbers on the US keyboard would not work at all.
• If a script has native digits but none of the currently shipping keyboard layouts use them, users won't be able to enter numbers either (e.g. Mongolian).
• If a script uses both native and ASCII digits, you need to choose which one you will support (e.g. Persian keyboard layout has ASCII only digits, while Persian standard keyboard layout has native only).
• The issue will still be present for languages that you don't localize the UI into.
• Bad luck for custom cultures and custom keyboard layouts.

Don't also forget another challenge in that solution which is supporting the lookup for characters outside of BMP (e.g. Osmanya).

[rudyhuyn]

Hi Miloush.

I will try to answer to some of your questions.

If a user has the US keyboard under Arabic input language and the UI/Region is in Arabic, both your sources would say Arabic and typing numbers on the US keyboard would not work at all

Unfortunately, keyboards don't share layout info with PCs, in fact, they don't even send the character code to the PC, but instead they send an ID about the position of the key. This is why if you type 'A' on an AZERTY keyboard plugged to PC with the keyboard layout set to QWERTY (us) the OS will think it's a 'Q', I would prefer keyboards to be smarter but the protocol was created decades ago.

In your example, the PC will receive Arabian letters, even if typed on a QWERTY (US).

If a script has native digits but none of the currently shipping keyboard layouts use them, users won't be able to enter numbers either (e.g. Mongolian).

Copy paste already manage number conversions to 0-9.

The issue will still be present for languages that you don't localize the UI into.

The calculator uses the region of your PC, not the OS/app languages, so if you select Hindi as your region, it will use Devanagari numbers

[miloush]

Thanks @rudyhuyn, but I meant US keyboard layout. I was trying to say that keyboard layout and input language are two different things. The same way you can add French keyboard layout to the English input language, the same way you can use US keyboard layout with the Arabic input language (it still produces Latin letters, feel free to try). The point is, it shouldn't matter what language or keyboard is being used, what matters is what the user inputs.

Hindi is not a region, and it is not available in the "Country or region" dropdown in Settings > Time & Language > Region. I assume what you meant is called "regional format" on that page. If you select Hindi there, the calculator will still accept ASCII digits only (again, feel free to try). There are indeed native digits on the Hindi keyboard layouts, but unfortunately you need to use Ctrl+Alt to get them and the calculator hijacks those for different purposes.

And by localization I meant creating the resw files.

[miloush]

Copy paste already manage number conversions to 0-9.

That is a good point indeed. The CopyPasteManager::ValidatePasteExpression contains a call to

// Get english translated expression
String^ englishString = LocalizationSettings::GetInstance().GetEnglishValueFromLocalizedDigits(pasteExpression);

so we can get the functionality you are suggesting by calling the same in KeyboardShortcutManager::OnCharacterReceivedHandler just before looking up the buttons. That way, all resw files can stay the way they are with ASCII digits, i.e. you don't even have to "support 2 locales".

Furthermore, we can get the functionality that I am suggesting by replacing GetEnglishValueFromLocalizedDigits implementation with a call to FoldString API with MAP_FOLDDIGITS. The LocalizationSettings already makes a call to GetLocaleInfoEx API anyway, so to be honest I don't understand why the GetEnglishValueFromLocalizedDigits uses it's own, inferior implementation rather than just calling the FoldString API.

[rudyhuyn]

@miloush: Use FoldString with MAP_FOLDDIGITS is a great idea, I really like the idea of the app basing this conversion on Unicode specs (we need first to write unit tests to be sure MAP_FOLDDIGITS supports all the characters already supported by the app).

One solution will be to not localize Key shortcuts (it's currently the case via .resw files) and only use 0-9 digits. Then in KeyboardShortcutManager we convert KeyCode received using MAP_FOLDDIGITS so all digits will be converted to 0-9 digits and will activate the key associated to this digit.

[grochocki]

@ivadham Thanks for filing this issue and to everyone else for the discussion on what might be happening here. We had the chance to review the pitch in more detail, and we think this might be a bug. Assigning to @joseartrivera to investigate further.

[joseartrivera]

Sorry for the delay on my response. While I believe this is technically a bug/oversight I think it would be nice to have a light-weight technical spec where we can discuss the implementation details further since this would likely be re-working a large portion of existing keyboard manager code.

[ghost]

This pitch looks like it has everything it needs for review. In the meantime, we'll keep this idea open for discussion so the community has the chance to provide feedback. Check out our New Feedback Process for more info on the user-centered process we follow for new feature development.

[ghost]

We reviewed the pitch and would love to explore this idea further! The pitch is a great start, but there are still some open questions. I am moving this issue into planning to iron out some of those details and I created calculator-specs/numeralSystems to track progress. A human will follow up with some feedback on your pitch shortly. Keep in mind that not all ideas that make it into the planning phase are guaranteed to make it to release. For more information on next steps, check out our spec workflow.