Unicode input news?

[Jiehong]

Hello,

As far as I understood it, we can use UC(n) to enter 4-hex-digits unicode symbols with the keyboard, but it is dependent on the OS. I wanted to understand how this works, and I think this Wikipedia article explains that well.

However, I would like to understand a bit better how the firmware tells the OS to input characters.

Couldn't the keyboard's firmware directly send unicode points, and the OS simply displays it as-is? Why is there a need to use a os-specific way to do that? (On my case, Linux, there isn't even a agreed-on way to do that anyway…).

I naively thought that having a new keymap would be to simply send the required unicode point for each key, or some sort of combinaison for modification keys.

Is the firmware stuck into only sending some keycodes that must be mapped as well in the OS?

I've got an ergodox with factory settings, but if I wanna use let's say bepo with it, does it mean it will work even if the OS is set to a QWERTY layout or should the layout also be switched to BEPO?

Perhaps the documention lacks a bit of information on this area, or I'm just too ignorant on this matter. On the latter case, you may have some links allowing me to understand this better?

P.-S: It's not really a bug report here, as it's mainly a question. Sorry if this isn't appropriate.

[fredizzimo]

Unfortunately the people that made the USB HID specification, decided to not allow Unicode for keyboard devices.

So the keyboard has to send codes representing physical keys. The operation system then has a keyboard layout, which does the translation from that physical key to the character displayed. So your operating system keyboard layout needs to support all the characters you want to display. The firmware lets you map them in whatever order or custom layout you want.

The same USB HID specification also specifies the Unicode page, which could be used to send any character to the OS. The problem with that is that no operating system that I know of implements it, so you would have to write your own driver. Another much less serious problem is that the encoding uses UCS-2, which can't represent all Unicode characters (only plane 0 is supported)

And I don't think any OS implements a "Magic Carpet Simulation Device" either.

[Jiehong]

Thanks for that specification.

So, basically, we have a list of pre-defined key with hexadecimal ids (from 00 to E7). Upon reception of such an id, the OS driver of your layout maps it to something, and you get that out.

Therefore, given a predefined layout (like QWERTY), changing the firmware allows you to change the position of the key by changing what key id is sent on each key. Single-key shortcuts are nice too.

Therefore, with a QWERTY keyboard, I would simply be unable to input ®, as the layout does not include this character.

Am I now correct, or did I miss something else?

[fredizzimo]

Yes, that's correct. You have at least four options

1. Use the UC macro as you already mentioned
2. Switch to another OS layout that supports all the characters you want
3. Make your own OS layout. On Windows you can use the Microsoft Keyboard Layout Editor. But your user logo indicates that you use Linux, and then the process is a little bit different on the distribution, but here are instructions for Ubuntu. On MAC you can use Ukelele.
4. On windows you could use Autohotkey, to make a sofware keyboard layout.

[Jiehong]

Thank you again for all your help!

If you think this kind of explanation is worth having, I can make a pull request, and try to put it somewhere. Otherwise, we can just close this ticket.

[fredizzimo]

I'll let @jackhumbert and @ezuk, decide if this should be documented somewhere, and what form of documentation it should be.

But if you ask me, then it should definitely be documented, with more detailed descriptions, than what I gave.

[fredizzimo]

While browsing the net I found the following https://github.com/kiibohd/KiiConf/issues/30. That goes in much more details about HID and Unicode. Haata is also looking for companies to join him in the effort of supporting Unicode. So if he hasn't already contacted you @jackhumbert, @ezuk, and @skullydazed, then maybe you should contact him.

[ezuk]

@fredizzimo @Jiehong - I am all for documenting this. Please do submit a PR updating the docs with this information, I think it's quite important and helpful. Thank you for the question! Once the PR is pending please feel free to close out this issue.

[Jiehong]

I'm closing as a PR is now underway

[alkeryn]

Sorry to necrobump but i can't compile using "UC(0x00c9)" or anything really the function is not defined

[drashna]

@Alkeryn It sounds like you don't have the unicode functionality enabled.

https://docs.qmk.fm/feature_unicode.html

[alkeryn]

And how am i supposed to enable it? ^^

[drashna]

By adding one of the following to the rules.mk for your keymap:

• UNICODE_ENABLE=yes
• UNICODEMAP_ENABLE=yes
• UCIS_ENABLE=yes

[alkeryn]

Thanks ^^

[heangfat]

Hello, I will start to make a big keyboard with about 500 to 600 keys with Unicode characters. I read documents Device Class Definition for HID and HID Usage Tables and understood that the range of Usage ID for key code report is from 0 to 0xFFFF (in HID Usage Tables Page 59), whose volume is similar to Unicode BMP; also the large range from 0xE8 to 0xFFFF is unused.

I prefer the 3rd option mentioned by @fredizzimo , making my own OS layout after defining my usage ID for key code report (say within the range 0x1000 to 0x1300). This is a feasible and very good way but it then still becomes OS-dependent. I have read QMK Unicode Support. As @Jiehong mentioned, UC(n) in UNICODE_ENABLE depends on OS. Does X(n) in UNICODEMAP_ENABLE also depends on OS?

There is another way, which uses both Keyboard/Keypad Page 0x07 and Unicode Page 0x10. But I am not sure whether it still supports boot protocol.

Is it possible to make this kind of big keyboard boot compatible (meaning all keys can work in BIOS and DOS before Windows/Linux/Macintosh is loaded)(given that corresponding fonts are equipped)? Or it must be OS-dependent?