search

NicoHood / HID

Bring enhanced HID functions to your Arduino!
http://www.nicohood.de
MIT License
2.36k stars 409 forks source link

Different keyboard layouts #22

Closed htho closed 2 years ago

htho commented 9 years ago

It would be nice to have multiple keyboard layouts available. This should be as simple as putting the definition of the _asciimap variable into a separate, layout depended, header.

My knowledge about key-codes is too limited to fork this repository myself. I also think in order to make this feature actually usable, similar to teensy, there should be a menu available in the IDE - And I don't have a clue how to alter the IDE-Menu-Structure. Simply using code from teensy is no option as the keyboard layout implementation for teensy seems to be too different from Arduino/HID.

NicoHood commented 9 years ago

Its on my TODO list. But since I dont know the different keyboard layouts I never had a deeper look into this. Also I think its not that important. Why not simply switch to an english keyboard? On my Windows for example I've installed German and english layout, I can switch. By the way on Windows you can switch with Control + Shift. Or Windows + Space. This way you could automatically switch forth and back very fast and silent.

I will keep it in mind. Feel free to contribute anything to this. I'll probably have a look though, shouldnt be too hard but its time consuming.

htho commented 9 years ago

Of course for expert users this is not a problem, but it sure is for novices. I also want to be able to use my devices on as many computers as possible and not to depend on a workaround.

Think of a scenario like this:

<me> Hey mom, this is the device I told you about.
<mom> Great! Where do I put it?
<me> Into the USB Port.
<mom> Is it the round one?
<me> No, thats for headphones... This one!
<mom> Ahh, I see!
<me> Anyway... you need to change the keyboardlayout first...
<mom> The what?

We will leave this little scene now before we start imagining how she tries to paint over the letters on the keyboard using an Edding...

I think I will contribute my solution when I implemented it.

NicoHood commented 9 years ago

https://github.com/PaulStoffregen/cores/blob/master/teensy/keylayouts.c https://github.com/PaulStoffregen/cores/blob/master/teensy/keylayouts.h

The easiest thing would be just to copy those definitions and make the Keyboard API work with it. If you just want a simple modification to switch Y and Z for example one could do it a bit simpler.

I decided to not work on that for now, rather try a full NKRO keyboard instead.

NicoHood commented 9 years ago

htho, have you seen the PR? You may want to try this out and give some feedback. I still havent tested it though.

htho commented 9 years ago

Hey @NicoHood , I havent tried it yet. Right now I am focussing on the hardware development. I'll give feedback when I am working on the firmware again.

NicoHood commented 9 years ago

@RingoM @htho You still interested in this? Version 2.4 now gives the option to use different layouts.

You can do this by calling

Keyboard.useLayoutGerman();

The patch was added here: https://github.com/NicoHood/HID/commit/5a13d971e72b78700f01327315a2e1efbd49c32f

Important Notes

Feel free to copy the US layout, modify it for you language and open a Pullup request

Please test this if you can.

htho commented 9 years ago

This a great step ahead. Now I am very busy writing my masters thesis (10 days left). I can't wait to try it. For my purposes the special characters are important too. I will contribute when I am done with my thesis and had a little time to relax.

NicoHood commented 9 years ago

No hurries. Maybe the code will change in those 10 days. I am finding a way to not type all layouts myself, maybe we can use the Teensy code here. Good Luck with your thesis ;)

Implementing this would be nice: https://github.com/RingoM/HID/blob/Layout-Test/avr/cores/hid/USB-Core/teensy_Keylayouts.h

NicoHood commented 9 years ago

There are 4 options on how to implement different layouts:

I'd like to know which one you like the most. I tend to switch to one of the 2 last solutions since I can mostly copy the Teensy code and avoid any further bugs. Also this makes it clear that you cant change the layout at runtime, there is always this guy who tries to abuse this and it just wont work ;)

Opinions?

htho commented 9 years ago

In fact it would be great to have an option to change the keyboard layout without needing to re-compile and upload the code. Until then any stable solution is sufficient. The boards.txt is not desired. I did not understand the last option. From a programmers point of view the current solution is very "arduinish", while not being able to change the layout at runtime with the same function is unintuitive. Therefore I prefer #define/#include which to the interested reader instantly makes clear this is a compile-time solution.

NicoHood commented 9 years ago

Example: You have to declare the used protocols here: https://github.com/NicoHood/IRLremote/blob/dev2/examples/Receive/Receive.ino#L34

The same would be for Keyboard layouts. Switching them at runtime takes tons of flash and wont be implemented.

I think there is a way to even provide a default instance, If you do not create a new one and overwrite it. But this needs to be tested. The point is, if we add this "feature" with templates you have to declare this in every sketch once. Not a big deal, but default examples wont work this way. But well thats the price. Or we create whole new names like GermanKeyboard.write('x') but I dont like that.

NicoHood commented 9 years ago

This could easily fix our problem without any complicated workarounds: https://github.com/arduino/Arduino/pull/3757

I'll wait until there is some progress in yes/no

NicoHood commented 9 years ago

I now adapted the Teensy Keyboard to the library. Its set to german layout right now. Syntax is as the original Teensy library, use TeensyKeyboard.write('a') for example. https://github.com/NicoHood/HID/commit/8d24749ec7eb3ab6add5f59d0828c540011624a6

The problem I am, currently having is that ESC and Backspace doesnt work. And I dont know why. these unicode wrappers are so complicated. I found a similar post in the forum, but the provided solution didnt help. The error occurs on german and english layout. https://forum.pjrc.com/threads/26783-No-support-for-sending-the-ESC-keycode-from-the-Arduino-keyboard-library

The other solution would be porting the whole Arduino ascii array to each language. I am not that happy to do that, at least not for other languages. The good thing about this idea is that the Arduino library uses less flash, so I'd prefer this if I really want to use a keyboard. The Teensy API is more flexible and has lots of definitions, also for those é keys and other stuff like that.

Maybe someone can help here (finding the teensy bug)?

RingoM commented 9 years ago

@NicoHood I just did a quick test with the code in my PR, which is essentially the same code you used with the teensy implementation and it seems that

TeensyKeyboard.press(KEY_BACKSPACE); TeensyKeyboard.release(KEY_BACKSPACE);

works just fine. I didn't have the time to install and test your code, but I have a strong feeling that it would also work with press and release.

In fact TeensyKeyboard.write() doesn't work with any keycode, so

TeensyKeyboard.write(KEY_A) doesn't work, but

TeensyKeyboard.press(KEY_A); TeensyKeyboard.release(KEY_A);

does.

That's what I understand from a quick reading on the teensyduino website: https://www.pjrc.com/teensy/td_keyboard.html

Keep up the good work!

NicoHood commented 9 years ago

That fixes the problem. Seems the Teensy Keyboard behaves different.

One problem left: The Teensy definitions for KEY_BACKSPACE are different that the official ones. It will overwrite those and the old wont work again. Should I only include the Teensy Keyboard if you use a #define USE_TEENSY_KEYBOARD or what? I cannot work around this, since it is a definition, and both APIs work different, but with the same makro names.

Please remind me to add this information (press+release instead of write) to the docs later if I forget this.

I will move those definitions (KEY_WHATEVER) to a general file, since the NKRO keyboard also uses the same keys. I'd also pledge for a multilayout patch for the original HID core, since the teensy is not that relyable in this case as you see. I want to keep it as option, but users who want to switch the layout should be able to use the standard API as well. Give me some time to work on this and also apply NKRO ;)

NicoHood commented 9 years ago

I tried to adapt the official keyboard API to another layout (german as example). Have a look at this commit: https://github.com/NicoHood/HID/commit/962b1a725fa3e5b19a54401eab1e3b2849b851c6

The problem here is, that english layout works with normal keys + shift. On a german keyboard for example you need to press ALT GR in order to get {[]} . So we need something like the SHIFT flag. But there is no option to set another bit in the array. >=128 is used for shift and the rest is used for normal keys. We could use uint16_t which will blow the Array size (I guess thats what the Teensy code does).

So for a german keyboard {[]}|@ are out of scope with this api. You can use the Teensy or better switch to english layout.

I also tried to sitch the keyboard layout dynamically before pressing a key. Ctrl + Shift for windows and for linux alt space. However this needs some delay and is not relyable. I think I will buy an english main keyboard then. Also I tried to set the keyboard layout for a specific keyboard on linux, but that changed it for both keyboards (arduino and usb keyboard). And not everyone is using linux too.

Any suggestions how to solve this?

RingoM commented 9 years ago

Arduino uses UTF-8 encoding, so when you send the letter 'b', you're actually doing

Keyboard.write(0x62);

Since the first 128 characters of UTF-8 correspond to the ASCII characters, arduino is using an ASCII lookup table to convert ASCII to HID keycodes.

Now, let's say you're trying to type the '€' sign, which is 0xe282ac, you're sending

Keyboard.write(0xe2); Keyboard.write(0x82); Keyboard.write(0xac);

The teensy code converts the UTF-8 encoding to Unicode and then the Unicode to keystrokes. Unicodes up to 127 are translated through an ASCII lookup table. Unicodes 128 -> 255 through an ISO 8859-1 lookup table. Other Unicodes (like the eurosign) are hardcoded.

So adapting the official API to german layout is not going to work without rewriting the entire API. I think arduino should import the teensy code. Although it's not perfect, the approach of converting UTF-8 to HID keycodes and allowing to switch layouts is better, according to me.

Arduino & UTF-8: http://playground.arduino.cc/Code/UTF-8 UTF-8 <-> Unicode conversion: https://en.wikipedia.org/wiki/UTF-8 UTF-8 & Unicode tables: http://www.utf8-chartable.de/ ISO 8859-1: https://en.wikipedia.org/wiki/ISO/IEC_8859-1 Windows keyboard layouts: https://msdn.microsoft.com/en-us/goglobal/bb964651

NicoHood commented 9 years ago

Sorry for the late response, but you are totally right. Thx for the explanation. I will keep the TeensyKeyboard as optional, foreign language keyboard. The Arduino Keyboard API does not allow to use a different Keyboard language.

NicoHood commented 9 years ago

Closing this now since Teensy now works. There will be no other (fast) solution yet. I think thats fine now since you have 3 keyboard you can choose from. 2 working even at boot time now.

Keyboard layout needs to be manually set in HID-Project.h right now. More documentation coming soon.

Thx for all the help. If you have tested this with success feel free to still post it here, even if this issue is closed.

NicoHood commented 9 years ago

I want to add here that I am probably going to implement the UK keyboard for the normal API. It seems that every key is pressable via shift. However this is still impossible for most other languages.

Edit fixed: https://github.com/NicoHood/HID/commit/b28d83b1601a0566844bc884131ceff6b886b30e

NicoHood commented 8 years ago

Reminder for myself: Maybe we can use namespaces to include this dynamically? (currently TeensyKeyboard is not available)

NicoHood commented 8 years ago

Just realized that we CAN use multiple modifiers for the keylayouts and that there IS enough space for 2 more bits. I will try to work on this tomorrow.

You can see it here: https://github.com/PaulStoffregen/cores/commit/f4723ec32285a1fdb67c244f32e36169a7a6477a

BTW: Consumer keys now also work with the keyboard (linux)

NicoHood commented 8 years ago

Now, let's say you're trying to type the '€' sign, which is 0xe282ac, you're sending Keyboard.write(0xe2); Keyboard.write(0x82); Keyboard.write(0xac);

Also if I do Keyboard.write('€'); ? I am wondering how i could differentiate between them properly. The keyboard API is getting more flexible soon, but also has more overhead (because of all virtual functions). However is better to maintain and fancier. Building HID bridges is then also easier.

https://github.com/arduino/Arduino/issues/2519#issuecomment-150910914

NicoHood commented 8 years ago

The guys who asked for this feature: Do you want to change the layout at runtime? I do not want to make it overcomplicated, one layout should be fine?

amoriello commented 8 years ago

You can imagine a factory like API e.g : 

   keyboard = KeyboardFactory(KeyboardLayout::english_us);
   keyboard.press()...

I think it would be great, but we have to consider impact on code size.

amoriello commented 8 years ago

Also, to support multiple layout at runtime, Keycode values should no be defines.

You made good progress in this direction using enums.

NicoHood commented 8 years ago

I've notices that most keycodes are the same for all layouts. actually just the ascii codes are country specific. Meaning a,b,c123!"£$% etc.

So the basic API keeps the same, just the print("I am a euro sign €"); can be changed.

I am probably going to make US default and you can add other layout on top of that. This way you can also change the layout at runtime, but only the used layouts + default US are compiled.

But I have to deal with some inheritance errors atm.

NicoHood commented 8 years ago

If you have any idea... https://gist.github.com/NicoHood/954b62efe66ca9747295#file-keyboard-h-L35-L36

Feel free to join the chat on freenode irc #arduino

NicoHood commented 8 years ago

First step was done: https://github.com/NicoHood/HID/commit/eff788e5405f47ce348c2f7cee2c47aab95c9efd

Requires: https://github.com/arduino/Arduino/pull/4027

If you look at the normal source file it is better visible: https://github.com/NicoHood/HID/blob/eff788e5405f47ce348c2f7cee2c47aab95c9efd/src/HID-APIs/KeyboardAPI.h

We now have more layers. KeyboardAPI Default/NKRO API (due to different HID reports) USB HW API (single/Multireport)

The language "packs" would now just use the base class pointer "KeyboardAPI" from one of the keyboards and use the Keycode functions to write raw data. With this technique you can code almost anything with the reports now.

I am not sure where to put the language packs so. As described above it could just access the data via the public functions (or via friends directly). It could also be a pointer inside the API which you can switch at runtime. Or it could be inherited in a layer above (or below??). Last option would be a static/fixed implementation.

However there is a lot more work that needs to be done to tear down the #define values into enums or arrays. So one header doesnt overwrite the other.

But I will fix the NKRO API first.

Tests appreciated!

amoriello commented 8 years ago

Wow,

You're quick. Making this patch rely on arduino/Arduino#4027 that isn't reviewed / integrated is a bit hardcore.

Yet, I will create a new branch on trustline to follow you, but I can't keep it on develop where cloners would fail to build because of non-void HID::SendReport (I don't want to expect developers to patch Arduino before build).

I'll digg into your library this week. I first have to understand the relationship between Keycode / Keyboard layout and printed characters. If you have any resources...

NicoHood commented 8 years ago

Keycodes are the raw definition of the usb protocol. Different keyboard layouts remap those. Which is annoying. but otherwise you would not be able to press ae oe ue on a german keyboard for example. And some keys are just flipped like y and z.

And it seems that there is no simple way to get rid of this since every keylayout is so different. sometime with shift modifier, alt gr and even many others.

Docs will follow. If i had updated them every time... wow i changed so much and will probably change in the future. Too bad we are between two versions and the wiki is somehow "floating".

The PR requirement is hardcore I know. But they will merge it for sure, because they like my work and know that its required if I need it :D Just wait for monday or maybe Tuesday.

NicoHood commented 8 years ago

The Teensy API requires to read the current pressed modifiers when a deadkey is used. Its not a problem to implement this, but this messes up the whole API and adds even more overhead. I dont want this.

Also what is a deadkey, do we need this? If we need it, we could use the DefaultKeyboard layer to read this data, meaning NKRO will be layout incompatible.

Also this code is quite a mess. The original Teensy code and the adaption I am trying to create. On top of this the IDE UTF8 encoding is not perfect. https://github.com/arduino/Arduino/issues/2519#issuecomment-150910914

So I dont really have the intention to implement this stuff. Its just... "dont use fucking other keylayouts". I just dont know how to implement this smart enough. On the other side there is now room for another library that can be placed on top of mine. So the limit is not set, its just not implemented by me.

And because the IDE handles UTF8 as shit it is hard to do something like Keyboard.press('€');. Since you never know what that should be. It just conflicts too much. Not saying its impossible.

So the frustrated nicohood is stopping work here for now. The only thing I imagine I could implement is a (fixed!) different ascii keymap with shift and altgr modifiers. This way you can implement the following keylayouts with all basic keys (€, german ae, oe and french e with hat are NOT possible then, but at least y and z, # " @ are placed at the correct position, aka first 128 ascii keys). This would work with most layouts. For example with a german keyboard this is NOT possible:

#define ASCII_60    GRAVE_ACCENT_BITS + KEY_SPACE       // 96 `

To work around this we could use shift AND alt gr together as special modifier sign. Then we can make a few more layouts compatible.

Another option would be to keep it as teensykeyboard again, with separate API, completely different from the normal API. But then we just can use the defaultkeyboardapi layer...

Any opinions? Anyone willing to implement this? I can share my dev code though.

amoriello commented 8 years ago

My 2cts: Character map is a mess, and character encoding a difficult problem, so this isn't an easy task.

C++11 standard introduces utf8 and utf16 encoded string as part of the language with u* prefix. http://stackoverflow.com/questions/6796157/unicode-encoding-for-string-literals-in-c11

That should solve the code source issue when you want to print(''€"), but it depends how the editor handles utf characters. Clearly, arduino editor fails short in this.

One way to go would be:

the Arduino print API cannot handle such task, it was designed to handle ASCII characters, and that's all. So if we want to solve this problem, we will have to break something.

If character encoding / code points are still a mystery, this is an helpful resource: http://www.joelonsoftware.com/articles/Unicode.html

NicoHood commented 8 years ago

Didnt read that text, but saw this line: "And all was good, assuming you were an English speaker."

The Teensy API supports uint8_t utf8 and converts them into a uint16_t utf8. so we can use both with the given code.

In the end you need a character map I think. But I will not do that now, I already spent too much time on that feature even though I will never use it.

NicoHood commented 8 years ago

Why dont we use the alt + numbers trick to press different UTF keys? This way we could be able to:

donid commented 8 years ago

I have created a new repository on GitHub that contains a working prototype which uses this library and UTF-8 to allow sending strings to a host that expects German language layout: https://github.com/donid/akuhell

Please consult the "readme" for more information.

NicoHood commented 8 years ago

This looks interesting. You did it all right, as you used the HID Project as backend. If the example keys in your sketch all work, then its a huge progress.

If you can pack this into a working Keyboard inherited class, that'd be perfect. Maybe something like a big lookup table would clean up the code more and support for other languages might also be simpler to add. If you have any progress, please let us know here. Thanks for sharing :)

schneirob commented 6 years ago

I learned a lot from this discussion implementing my idea of sending the keycodes from a raspberry pi to the arduino via i2c, so I am doing most of it in python and not on the arduino (https://github.com/schneirob/i2ckeyboard). So thank you for discussing!

Allow me to comment something I stumbled upon (and haven't implemented, yet):

One thing to keep in mind, when designing a new lookup table / keymap feature that there are characters that are composed by multi key action and require key releases in between (e.g. ÈÉèéÀÁàá). When running a German keyboard Layout you produce those by pressing KEY_LEFT_SHIFT, KEY_EQUAL, Release All, (KEY_LEFT_SHIFT), KEY_E/KEY_A. Or with Grave âêôÂÊÔ. As I am using key event codes coming in from evdev, the idea is to have a lookup table to map the keys from the keyboard layout to the keys to press and have another table for the character lookup, that then has a list of keys to press for a certain character. To solve the release issue, one could have a keycode that is later caught and used to release all keys (a virtual "release key".

Just things that came to my mind, hope it helps!

paulens12 commented 5 years ago

sorry but i've never in my life seen a keyboard that somehow physically, or in its firmware, "supports" a specific layout. All keyboards send the same scan code for each corresponding key, and the OS then decides which character to assign to that scan code depending on the language you set.

What you're suggesting: how would this even work? You plug the keyboard in, and the OS says your keyboard layout is US English, but the keyboard instead writes some unrelated foreign characters? Wut? What if you want to actually switch to US English?

A suggestion for your mom: next time you install Windows, don't forget to set the right keyboard language for her. That way you won't have to build her specialized keyboards with specialized firmware that do the layout conversion for her OS.

NicoHood commented 5 years ago

@paulens12 In non english environments it is very common that you have special characters. And you are right, that the keycodes are always the same, that you send. But the keyboard API abstrcts that layer. You dont specify which key was pressed, instead you want an API that tells you wich character should be written on a specific layout.

The requirement is totally legit, but not trivial to implement. Not in a way that would make it general enough, or in a way I was motivated to do so.

paulens12 commented 5 years ago

Don't worry, you don't have to explain me how international keyboards are different - I have used 6 different keyboard layouts regularly (for 5 different languages, although I don't speak them all, sometimes it's cheaper to import laptops from certain countries).

But can't we just send scan codes using your API? I really don't get this whole fuss about "layouts". This concept should not exist at all in keyboard controller firmwares. Either use scancodes, or for easier configuration, add #defines with all US QWERTY keys, e.g.

define KEY_A 0x00

define KEY_ENTER 0x00

and so on (I don't know the actual scan codes so I just used 00 everywhere) I really don't see the point in any further abstraction. Even game engines handle user input this way :)

Sorry if I'm saying the obvious - I haven't had time to get more in depth with your library yet.

From: Nico notifications@github.com Sent: Saturday, January 26, 2019 8:57:40 PM To: NicoHood/HID Cc: paulens12; Mention Subject: Re: [NicoHood/HID] Different keyboard layouts (#22)

@paulens12https://github.com/paulens12 In non english environments it is very common that you have special characters. And you are right, that the keycodes are always the same, that you send. But the keyboard API abstrcts that layer. You dont specify which key was pressed, instead you want an API that tells you wich character should be written on a specific layout.

The requirement is totally legit, but not trivial to implement. Not in a way that would make it general enough, or in a way I was motivated to do so.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/NicoHood/HID/issues/22#issuecomment-457856081, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADg7zfZvoFWQUdNmuweNHQq9JltYpb-Kks5vHKUkgaJpZM4D_lhH.

NicoHood commented 5 years ago

It is not that simple. Some keys can only be reached with Shit or Alt GR combinations. This required some coding overhead that is not handled yet. And thats the problem. Please go ahead and write some code...

paulens12 commented 5 years ago

ummm, what's keeping the user from pressing shift and alt-gr? Why does it require special handling when it's actually the same as writing A by pressing shift+a?

NicoHood commented 5 years ago

Because no user is pressing the key, the API simulates those keystrokes. I guess you totally miss the point. I will stop discussing here, as this will not bring us any further.

MichaelDworkin commented 5 years ago

Leonardo international keyboard support MultiLanguage library

Keyboard library for Arduino for all languages and keys mapping

This library allows an Arduino board with USB functions to act as a keyboard. The following languages are integrated: English, German, Russian, French, Greek, Hebrew, Italian, Georgian

NicoHood commented 2 years ago

Different keyboard layouts are now supported.

swiersyoram commented 1 year ago

@NicoHood If you have time, could you please explain why paulens12 is wrong and you actually do need keyboard layouts in firmware. Cause I was thinking the same thing, if the keyboard layout is set by the OS, why would you also need different layouts in firmware. I don't see need for the api to simulate keystrokes.