search

machinelevel / sp425-crazy-cow

Add crude keyboard-lettering function to eink tablet
MIT License
77 stars 6 forks source link

Would like a little more info on how crazy this cow is ! #1

Open DariusMofakhami opened 4 years ago

DariusMofakhami commented 4 years ago

Hi, Your project sounded awesome to me, but there are some details I don't understand. If i understood right, you need to launch your remarkable differently than with the usual UI ? Or can you just switch to the keyboard typing mode by plugging a keyboard ? My main point is trying to understand to what extent installing your code on a rM device will change the original writing experience with a pencil ? Thanks a lot for this crazy project ! Darius

machinelevel commented 4 years ago

Hi Darius! Thanks for your note.

you need to launch your remarkable differently than with the usual UI?

Nope, you start the reMarkable up just normally.

Or can you just switch to the keyboard typing mode by plugging a keyboard?

Yes, exactly. I used it this morning, actually with a cheap USB wireless keyboard. Just plugged in the USB and started typing. That was the plan, and so far it seems to be working. :]

My main point is trying to understand to what extent installing your code on a rM device will change the original writing experience with a pencil?

No change at all. This morning I typed characters and also wrote notes next to them.

One small quirk to know: the "backspace" feature uses the reM Undo tool, so if you type and hit backspace it's fine, but if you pencil and then immediately backspace, it'll undo some of your pencil art.

Thanks a lot for this crazy project !

You're very welcome! I hope you enjoy it. So far I'm finding it nice to use.

DariusMofakhami commented 4 years ago

Ok ! Good to hear ! But so what kind of keyboard do you plug in ? I tried with my computer keyboard, doesn't work, but I guess this is because rM tablet cannot power on the keyboard... What does the usb wireless keyboard u r using looks like ? I'm pretty excited about making the cow getting crazy :p

machinelevel commented 4 years ago

Hm, one thing: it requires a USB OTG ("on the go") adapter to convert the keyboard's USB to the MicroUSB of the reM. I've successfully used both this ($5.59) and this ($4.99 for 5 of them).

If that connector's correct, then one keyboard I've used successfully is this ($11.69), and I've also used an older 2.4GHz USB wireless keyboard by plugging the 2.4GHz receiver into the OTG adapter.

DariusMofakhami commented 4 years ago

So I've followed the install instruction. I've plugged in a USB bluetooth receiver via the OTG USB and try the associated keyboard to type down stuff ! But nothing happened :( I'm using the wireless microsoft keyboard 2000 (working on my computer which is windows 10) and I know it is connected as I can control the page swipping with the keyboard left and right arrows. But clicking on a letter does not affect the current notebook. May it be the keyboard which isn't simple enough ? Or the fact it was working with windows and not Mac or Linux ? (Unix system ?) Would you have any guess ? Or may it be that I have the custom remarkable-hacks from ddvk installed on my rM ? (https://github.com/ddvk/remarkable-hacks)

machinelevel commented 4 years ago

Hi there, Thank you for so much detailed info! As far as I know, you're the first person to try this besides me, and there's a lot we can do to try to debug this.

First of all, the left/right arrow keys were a great find. The reM built-in software supports those, so that's the perfect way to test that the keyboard is connected and supported.

As a next step, I've just pushed a new branch (complete with binary) specifically to solve this, called 1-device-testing. That version just does a ton of diagnostic printing.

Here's how to try it:

  1. ssh into your device, and type systemctl stop crazy-cow to stop the service.
  2. Download the debug version crazy-cow-debug and copy it to your device.
  3. In your ssh window, type ./crazy-cow-debug and you should immediately output like this: 
    ]] at line 586 in main
]] at line 579 in initialize
]] at line 581 in initialize

    If no keyboard is plugged in, you should see this every few seconds:

    .. at line 550  (no keyboard)
]] at line 503 in find_devices
  .. at line 509 in find_devices
  .. at line 509 in find_devices
  .. at line 509 in find_devices
  .. at line 509 in find_devices
  .. at line 509 in find_devices
  .. at line 512 in find_devices
   .. found wacom dev platform-21a4000.i2c-event-mouse
  .. at line 516 in find_devices
Connected to pen input device /dev/input/by-path/platform-21a4000.i2c-event-mouse.
  .. at line 509 in find_devices
.. at line 554  (still no keyboard)

    When a keyboard is plugged in, you should see something like this, just once:

    ]] at line 547 in do_main_loop
.. at line 550  (no keyboard)
]] at line 503 in find_devices
  .. at line 509 in find_devices
  .. at line 509 in find_devices
  .. at line 509 in find_devices
  .. at line 509 in find_devices
  .. at line 526 in find_devices
   .. found keyboard dev platform-ci_hdrc.0-usb-0:1:1.0-event-kbd
Connected to keyboard device /dev/input/by-path/platform-ci_hdrc.0-usb-0:1:1.0-event-kbd.

    After that, with a notebook open and a pen selected, if you press the letter "A" on your keyboard you should see this output:

    ]] at line 547 in do_main_loop
]] at line 547 in do_main_loop
]] at line 547 in do_main_loop
]] at line 547 in do_main_loop
]] at line 413 in handle_event
  .. got key event type:1 code:30 val:1
]] at line 486 in handle_event
  .. character to type:a
]] at line 547 in do_main_loop
]] at line 547 in do_main_loop
]] at line 547 in do_main_loop
]] at line 413 in handle_event
  .. got key event type:1 code:30 val:0
]] at line 547 in do_main_loop
]] at line 547 in do_main_loop

    ...and should see a lower-case "a" appear right next to the pen tool: image

If this fails...

All of those printed lines correspond to lines in main.cpp like this one, so at least we can see what's failing. It might also be worthwhile to try ls /dev/input/by-path/ in your ssh window, to see what's there and if plugging in the keyboard changes.

To start the non-debug version, you can just systemctl start crazy-cow in your ssh window again.

Please let me know what happens, and thank you for testing!

DariusMofakhami commented 4 years ago

Ok so here is what happened : I've copied and launched crazy-cow-debug (note that I had to chmod +x crazy-cow-debug before being authorized to run it) And the predicted message kept appearing regularly until I plug the usb bluetooth receiver of the keyboard in my rM tablet. Then I got the confirmation that a keyboard was connected : 

  ]] at line 547 in do_main_loop
    .. at line 550  (no keyboard)
    ]] at line 503 in find_devices
      .. at line 509 in find_devices
      .. at line 509 in find_devices
      .. at line 509 in find_devices
      .. at line 509 in find_devices
      .. at line 509 in find_devices
      .. at line 512 in find_devices
       .. found wacom dev platform-21a4000.i2c-event-mouse
      .. at line 509 in find_devices
      .. at line 509 in find_devices
    .. at line 554  (still no keyboard)
  ]] at line 547 in do_main_loop
    .. at line 550  (no keyboard)
    ]] at line 503 in find_devices
      .. at line 509 in find_devices
      .. at line 509 in find_devices
      .. at line 509 in find_devices
      .. at line 526 in find_devices
       .. found keyboard dev platform-ci_hdrc.0-usb-0:1:1.2-event-kbd
  Connected to keyboard device /dev/input/by-path/platform-ci_hdrc.0-usb-0:1:1.2-event-kbd.
      .. at line 509 in find_devices
      .. at line 512 in find_devices
       .. found wacom dev platform-ci_hdrc.0-usb-0:1:1.1-event-mouse
      .. at line 509 in find_devices
      .. at line 526 in find_devices
       **.. found keyboard dev platform-ci_hdrc.0-usb-0:1:1.0-event-kbd**
      .. at line 509 in find_devices
      .. at line 509 in find_devices
      .. at line 509 in find_devices
      .. at line 512 in find_devices
       .. found wacom dev platform-21a4000.i2c-event-mouse
      .. at line 509 in find_devices
      .. at line 509 in find_devices
  ]] at line 547 in do_main_loop
  ]] at line 547 in do_main_loop
  ]] at line 547 in do_main_loop
  ]] at line 547 in do_main_loop

But after that, opening a new notebook and typing 'a' or whatever other key did not cause any additional message to pop up in the terminal, nor any letter on the notebook. Still, the basic input as you said were working : keyboard arrows, and the escape and home keys that would act like the rM1 central button.

Additionnaly, when I look in ls /dev/input/by-path/ without the keyboard I get :

platform-21a4000.i2c-event-mouse  platform-21a8000.i2c-event
platform-21a4000.i2c-mouse        platform-gpio-keys-event`
And after pluggin the keyboard I get : 
`platform-21a4000.i2c-event-mouse            platform-ci_hdrc.0-usb-0:1:1.1-event-mouse
platform-21a4000.i2c-mouse                  platform-ci_hdrc.0-usb-0:1:1.1-mouse
platform-21a8000.i2c-event                  platform-ci_hdrc.0-usb-0:1:1.2-event-kbd
platform-ci_hdrc.0-usb-0:1:1.0-event-kbd    platform-gpio-keys-event

So i'm pretty sure the keyboard recognition is okay, but no letter is interpreted by the crazy cow ? :(

machinelevel commented 4 years ago

That's good info... clearly the keyboard device is being opened with no problem, so it may be the case that something else is intercepting the key events so crazy-cow doesn't get them.

Here's something to try: ls -l /dev/input/by-path/ ...should show you the device aliases. In my case, I can see that my keyboard is being mapped to event3...

lrwxrwxrwx    1 root     root             9 Sep 18 13:49 platform-21a4000.i2c-event-mouse -> ../event0
lrwxrwxrwx    1 root     root             9 Sep 18 13:49 platform-21a4000.i2c-mouse -> ../mouse0
lrwxrwxrwx    1 root     root             9 Sep 18 13:49 platform-21a8000.i2c-event -> ../event1
lrwxrwxrwx    1 root     root             9 Sep 18 17:10 platform-ci_hdrc.0-usb-0:1:1.0-event-kbd -> ../event3
lrwxrwxrwx    1 root     root             9 Sep 18 17:10 platform-ci_hdrc.0-usb-0:1:1.1-event -> ../event4
lrwxrwxrwx    1 root     root             9 Sep 18 13:49 platform-gpio-keys-event -> ../event2

...so then we can use lsof to see who else has it open, like this: lsof | grep /dev/input/event3 (replacing event3 with whatever your keyboard is mapped to) ...in my case, I see this:

systemd-l 174                 root   16u      CHR      13,67      0t0      14004 /dev/input/event3
xochitl   209                 root   23r      CHR      13,67      0t0      14004 /dev/input/event3
QEvdevTou 209 220             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
LoadingTh 209 226             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
QQmlThrea 209 227             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
Qt\x20bea 209 228             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
QDBusConn 209 229             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
QThread   209 232             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
QQuickPix 209 239             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
QNetworkA 209 263             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
DocumentW 209 269             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
QThread   209 270             root   23r      CHR      13,67      0t0      14004 /dev/input/event3
crazy-cow 323                 root    4r      CHR      13,67      0t0      14004 /dev/input/event3

This shows every process which found and opened the attached keyboard. In that list, I see xochitl (that's the reM built-in UI app) and I see crazy-cow, and then a bunch of others I don't know much about, but which don't seem to be causing any trouble.

If your list has items in it I'm missing, what are those?

DariusMofakhami commented 4 years ago

Ok nice ! May be we'll find out the problem ! When I ls -l /dev/input/by-path/ here is what I got : 

lrwxrwxrwx    1 root     root             9 Sep 18 12:13 platform-21a4000.i2c-event-mouse -> ../event0
lrwxrwxrwx    1 root     root             9 Sep 18 12:13 platform-21a4000.i2c-mouse -> ../mouse0
lrwxrwxrwx    1 root     root             9 Sep 18 12:13 platform-21a8000.i2c-event -> ../event1
lrwxrwxrwx    1 root     root             9 Sep 19 13:13 platform-ci_hdrc.0-usb-0:1:1.0-event-kbd -> ../event3
lrwxrwxrwx    1 root     root             9 Sep 19 13:13 platform-ci_hdrc.0-usb-0:1:1.1-event-mouse -> ../event4
lrwxrwxrwx    1 root     root             9 Sep 19 13:13 platform-ci_hdrc.0-usb-0:1:1.1-mouse -> ../mouse1
lrwxrwxrwx    1 root     root             9 Sep 19 13:13 platform-ci_hdrc.0-usb-0:1:1.2-event-kbd -> ../event5
lrwxrwxrwx    1 root     root             9 Sep 18 12:13 platform-gpio-keys-event -> ../event2

So you see my bluetooth receiver is connected to a keyboard and a mouse (which I don't switch on) But more interestingly I think, there are two inputs for the keyboard. So in my case, event 3 and event 5. And crazy-cow seems to be connected to event 5 only : When I lsof | grep /dev/input/event3 I got 

systemd-l 191                 root   16u      CHR      13,67      0t0      23083 /dev/input/event3
xochitl   666                 root   23r      CHR      13,67      0t0      23083 /dev/input/event3
QEvdevTou 666 670             root   23r      CHR      13,67      0t0      23083 /dev/input/event3
LoadingTh 666 676             root   23r      CHR      13,67      0t0      23083 /dev/input/event3
QQmlThrea 666 677             root   23r      CHR      13,67      0t0      23083 /dev/input/event3
Qt\x20bea 666 678             root   23r      CHR      13,67      0t0      23083 /dev/input/event3
QDBusConn 666 679             root   23r      CHR      13,67      0t0      23083 /dev/input/event3
QThread   666 680             root   23r      CHR      13,67      0t0      23083 /dev/input/event3
QNetworkA 666 683             root   23r      CHR      13,67      0t0      23083 /dev/input/event3
QQuickPix 666 685             root   23r      CHR      13,67      0t0      23083 /dev/input/event3
DocumentW 666 778             root   23r      CHR      13,67      0t0      23083 /dev/input/event3
QThread   666 779             root   23r      CHR      13,67      0t0      23083 /dev/input/event3

And when I lsof | grep /dev/input/event5 I got 

systemd-l 191                 root   17u      CHR      13,69      0t0      23092 /dev/input/event5
xochitl   666                 root   24r      CHR      13,69      0t0      23092 /dev/input/event5
QEvdevTou 666 670             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
LoadingTh 666 676             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
QQmlThrea 666 677             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
Qt\x20bea 666 678             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
QDBusConn 666 679             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
QThread   666 680             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
QNetworkA 666 683             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
QQuickPix 666 685             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
DocumentW 666 778             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
QThread   666 779             root   24r      CHR      13,69      0t0      23092 /dev/input/event5
crazy-cow 828                 root    5r      CHR      13,69      0t0      23092 /dev/input/event5

And here we see crazy-cow in the end of the list. May be event 5 is not the good one and I should find a way to redirect crazy-cow toward event 3 (or at least have it work with both event ?) Thanks for helping me out with that ! I feel like I'm begining to understand a bit more what your code is doing ^^'

machinelevel commented 4 years ago

Aha, clearly you're right. Somehow your setup is supplying two keyboards, and only one works.

  1. The immediate workaround I suggest is to, if possible, simplify your hardware setup so that only a single keyboard is added.
  2. I'll also explore the code change to support multiple keyboards. While it seems simple (and would be on a laptop), I want to be very (very) careful to avoid extra battery drain. Especially with eInk devices, that's what prevents a simple "polling devices" model. I'm sure there's a good way to do this, just need to try it and test it well.

With your current setup, is there an easy way to simplify it down to a single keyboard device?

machinelevel commented 4 years ago

Update: I've been able to connect two keyboards and exactly reproduce this issue. Looking into a clean fix...

DariusMofakhami commented 4 years ago

Cool, thanks ! About the workaround, because the debug version was giving interesting info, I tried again with a non-bluetooth USB keyboard (so basically the keyboard of my PC) And in fact, the rM do detect it (and I guess is able to power it) and this keyboard appears to be a single keyboard for the rM tablet. So once I launched crazy-cow, I become able to type letter down, as expected ! (One problem that I had however was with the backspace key, which do push back the cursor one letter back, but did not trigger the undo button.) I think that might be usefuul for other users to have a fix for the two keyboards issue to allow more keyboards to work, and so more people to be able to use your code. But at least, with this detailed issue, they will have a chance to now why their keyboard doesn't work. Now I have a lot of other feature requests hehe ! But I bet it will be cleaner if I write new issues ! And also, do you mind if I speak about your project on reddit in the reMarkableTablet page ? I feel like it could interest a lot of users !

machinelevel commented 4 years ago

To finish off this item, I've done a test implementation for multi-keyboard support. You can try the binary now, in the 1-multiple-keyboards branch of this project. It's not very well tested yet, but seems to do the trick.

oricgn commented 3 years ago

Hi machinelevel,

Hm, one thing: it requires a USB OTG ("on the go") adapter to convert the keyboard's USB to the MicroUSB of the reM.

I tried SAUX USB C to USB 3.1 OTG without success.

Regards Oliver