[Enhancement] keyboard control

[rhysmorgan134]

Hopefully this isn't too big of a job. But would be great to have the same keyboard functionality as open auto pro, I know myself and many others interface with the steering wheel controls via an Arduino running as a key board input. Would be extra helpful if it had the same mappings as OAP

[rsjudka]

I was just looking through Qt API and it seems pretty straightforward.

What kind of mapping does OAP? Does it have anything in addition to what the regular openauto has? It would be helpful if you gave me a list of all the different shortcuts you need/want and I can add the appropriate actions to ia.

[rsjudka]

I've been working on the key-bindings branch to allow you to enable the buttons in OpenAuto and to set custom shortcuts for elements in ia. Right now I've added shortcuts for opening each of the tabs, and for increasing and decreasing the volume.

Any other shortcuts you feel would be helpful?

[rhysmorgan134]

Sorry for the late response, been manic with work. Those shortcuts pretty much cover it, looking forward to testing. Only extra that I can think of, it's not quite a shortcut, but perhaps switching night/day in android auto, and simultaneously switching dark to light mode in ia? Would be great to be able to bind these and the other shortcuts to gpios so they can be linked to headlight status/solar sensors.

[robert5974]

With the Android auto app, the developer settings has the option to let day/night be phone controlled. What is the advantage of hooking it into the sensor of the vehicle? Just curious. Is it to sync it up with perhaps the headlights turning on?

[rhysmorgan134]

Yeah that would be the plan, ie when headlights are on, then night mode, when headlights are off then switch to day mode.

The bash script is working really well for install, will do a pull request at some point next week

[rsjudka]

Yeah the day/night mode switching is something I've had on my backlog for the longest, I just have a feeling it's gonna involve me digging into aasdk so I've been avoiding it :p the plan for that is to allow dark mode to be synced with open auto, where if either one changes the other will be toggled too.

I did add the keyboard shortcut for toggling dark mode, still need to look more into how GPIO pins can be assigned to various actions. I have some ideas for allowing either keyboard or gpio to be mapped, when I get some time this week I'll see how that goes.

[rsjudka]

@rhysmorgan134 just pushed the gpio shortcut functionality to the key-bindings branch! Would you be able to test it out and let me know how it works for you? I was able to wire up a simple button to test with it, but I'm curious to see how it works with whatever setup you got.

Also, the day/night mode switching will probably be worked on in a different branch, so keep an eye out for that!

Oh just to note, you'll need to make sure your gpios are setup prior to launching ia. Because of the complexities that come with setting up gpio (and potentials dangers if done wrong) I want to make the user sets it up based on their configuration, so that ia will simply respond to the gpio signals.

[rsjudka]

day/night mode is now in this branch (that feature was merged to develop and then key-bindings branch was updated to develop)! Make sure to recursively pull all the submodules too because there were some openauto changes required to make this work.

You'll need to go into the developer settings in Android Auto and set night mode to car controlled if you want it to work just an fyi

[rhysmorgan134]

Apologies for the lack of feedback, lockdown and childcare/works given me no time to get on this.

Just successfully built it, and now need to set up the app launcher etc, then will give it a full test in the car.

@rsjudka would you be ok with me doing a YouTube vid on it? Would be some good exposure. Not a problem at all if not.

[rsjudka]

no worries! times have definitely been weird

let me know how it goes and if you have any troubles with the gpio triggers

and sure :)

[rhysmorgan134]

Awesome, thanks.

I can't believe how much you have done over the last month, it's really looking good. I will have to have a play to figure out all the new stuff!

[rsjudka]

ah shoot looks like theres some instability with the recent merge from develop

i think i need to rework how events are handled because the shortcut signals (gpio included) seem to be clashing with the other signals

[rhysmorgan134]

Yeah, I had an early play with it, and for me if I try and set up the shortcut, I get nothing when pressing the box, not sure if that's the same issue that you are referring to (in dark mode the box is barely visible too)

I wasn't going to mention on here until I have had a proper play with it though, as I may of just missed something

-------- Original message -------- From: Robert Stanley Judka notifications@github.com Date: 30/05/2020 00:18 (GMT+00:00) To: rsjudka/intelligent-auto intelligent-auto@noreply.github.com Cc: rhysmorgan134 rhys1802@hotmail.co.uk, Mention mention@noreply.github.com Subject: Re: [rsjudka/intelligent-auto] [Enhancement] keyboard control (#25)

ah shoot looks like theres some instability with the recent merge from develop

i think i need to rework how events are handled because the shortcut signals (gpio included) seem to be clashing with the other signals

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/rsjudka/intelligent-auto/issues/25#issuecomment-636232562, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIZHOGXAWZXM2LCM3USLWZ3RUA7C7ANCNFSM4MGZXZGQ.

[rsjudka]

Hmm they're supposed to be faint, but I wonder if I'll need to up the contrast to deal with different displays.

No that would be a separate issue. Do you already have the gpio setup? Can you test with a simple python script to see if the button gets triggered there?

[rhysmorgan134]

Sure I'll take a look tomorrow, it's early morning here. I'm not even sure if I could get that far, as on the shortcut page I literally can't seem to access anything as it all seems unresponsive (again I only had a very brief look, so will give it another go over)

[rsjudka]

ah shoot looks like the scaling feature is causing the UI to occasionally hang with gpio triggers because of the way the events are now processed (might be the issue you're seeing)

Try using the commit before that (fcc33e9) and hopefully you can get going with it! I'll try getting a fix in for that asap

[rsjudka]

scratch that, was able to fix the issue so you should be good to use the latest on the key-bindings branch!

[rhysmorgan134]

still the same, I have attached the above gif that will hopefully show the issue, I think we may be on different issues

[rsjudka]

are you able to set keyboard shortcuts at least?

Are you able to verify your gpio buttons are triggering any event on your pi? Maybe with a simple python script or something.

I used this little script for testing:

from gpiozero import Button

b = Button(2)

enable_print = True
while True:
    b.wait_for_press()
    print("pressed")

not sure if you saw this also..

Oh just to note, you'll need to make sure your gpios are setup prior to launching ia. Because of the complexities that come with setting up gpio (and potentials dangers if done wrong) I want to make the user sets it up based on their configuration, so that ia will simply respond to the gpio signals.

[rhysmorgan134]

Yeah, my bad, I have got it rolling, works really well!!

One thing that would be nice, and it does kind of work, but say I have one button on my dash. I want to press that button to switch between pages, if in the settings page I set them all to the same gpio, would it be possible to toggle between the pages? Even if it was just between two would be handy to have.

I have tested with 2 shortcuts (app launcher and android auto) set to the same gpio pin, and it does sometimes work, sometimes it happens a bit sporadically.

Thanks for all the work you are doing!

[rhysmorgan134]

Also, maybe there is a way, but I can't figure out how to clear a key binding.

[rsjudka]

Yeah you probably don't want to set the same gpio for different shortcuts, but I can totally add a "page toggle" shortcut!

And yeah, to clear a shortcut you can click/tap and hold for about half a second

[rhysmorgan134]

Ok, I do have a really strange problem.

So setting it up on a spare pi, there’s no issue at all.

When I do it in the car, I got to the settings page, press the launcher input box, press the physical button in the car, it correctly inputs the right gpio number, but then it will never actually activate. Checking my app, that is logging that the button has been pushed, and it does pick it up in the settings screen. So the gpio is configured correct.

Is there any logging I can turn on or check?

[rsjudka]

Just added the page cycle shortcut!

And added some additional logging to help figure out whats going on with your issue. When you get the time to test again, let me know what you see.

[rhysmorgan134]

Thanks alot, I'll give it a test, may not be for a few days though.

Been using ia daily now in the car, and it's performing perfectly 👍

[rhysmorgan134]

Ok I gave it a test in the car with the extra logging, and I am indeed getting the log of

GPIO Changed GPIO inactive

both happen instantaneously upon releasing this button. This is the only difference that I can think of that my cause this. When shorting the wire across the pins on my test pi, I get a group of about 10-15 gpio changed-gpio inactive logs (is there a debounce on the input?)

I have tested a dark mode toggle one, and thats working ok.

On the note of darkmode, would it be possible to have it as input high = dark mode, low = light mode? The toggles a bit problematic when it comes to knowing the initial state of the canbus messages on the car, if that makes sense

[rsjudka]

did you see a log of [Shortcut][INFO] gpio active low is ...?

So maybe I made some bad assumptions, but how are your gpio set up? How ia handles them is basically registering all gpios found in the sysfs (and saving off their active low value) at startup, and then watching those files until the state is the active low state.

Hmm i would need to think a bit more on how to handle that. Im not sure I would want to consider that a "shortcut" since it would require mapping the inputs instead of just taking action on the input (if that makes sense)

[rhysmorgan134]

That's from one button press (sorry for the glare, we had some rare sun)

22 is one I'm testing for night mode 5 is the one that is currently not working

But there was

[shortcut]gpio active low is 48

That was on start up

This is my nodejs configuration using onoff which is working for functions in my app, will post separately the config from the pi

home = new Gpio(5, 'in', 'rising', {debounceTimeout: 10}

[rsjudka]

hmm i would have assumed there would be 2 "gpio changed" messages (one for the button press and one for the release)

is that what the debounceTimeout is for? Could you try unsetting that and seeing if that works?

I might just need to read up more on gpio lol

[rhysmorgan134]

The button is switching via ground, so the normal button state is high, you press the button, it goes low, when released, goes back to high.

The debounce is there just to say the button needs to be pressed for 10ms before emitting that the pins changed. Without it, you get terrible on off states each lasting a millisecond or so until the button is firmly pressed. In general you never run an input without a debounce.

I will relay the button pressed event via my app to a separate gpio that switches low to high, to see if that resolves it

[rsjudka]

hmm well thats weird then i definitely would have expected to see 2 gpio events in your log

i designed the gpio "watcher" to work for both default states (high and low) so that it really didn't matter how you have it configured. In your app, are you seeing both "press" and "release" triggers?

[rhysmorgan134]

On my app I'm solely looking for a rising edge, and then reacting to that.

So whether it's going low-high-low or high-low-high I still react in the same way. To be fair, the fact it works linking the input to ground should been your codes fine, but I think it's worth a try, it's pretty easy for me to do it.

[rsjudka]

Are gpio 5 and 22 different in any way?

[robert5974]

Which naming convention are you referring to? Physical number or BCM number? Easiest way to determine is pinout.xyz.
if you mean physical pin 5 then that one is: SCL (i2c Clock) is one of the i2c pins on the Pi. SCL includes a fixed, 1.8 kohms pull-up to 3.3v, which means this pin is not suitable for use as a general purpose IO where no pullup resistor is desired.

If you mean BCM pin 5 then: it's a general purpose pin. Physical pin 22 and BCM pin 22 are both general purpose.

[rhysmorgan134]

Yeah I am going by BCM numbers as they appear in the app so

BCM 5 or physical 29

About to test relaying it to a software controlled gpio, will let you know

[rhysmorgan134]

OK, looks like it was my fault, I had my app set for just rising edge detection, changed to both and now its all good, now getting the proper logging of both states.

One thing thats not working though is the page cycle, I believe its just due to missing a debounce, one press makes it switch multiple times, not sure if you know how they work, but when you have a physical button, and you make the contacts, theres a split seconds where the contacts are not fully made, creating a pulsing effect until the button is firmly pushed. The 10 milliseconds I use in my app has been pretty spot on.

Lastly just another note on the day night toggle, if its tricky to implement state detection, would a seperate shortcut for each be possible? With it all plumbed up now, its proving pretty hard to set up, due to the reasons highlighted above, but also not knowing what state IA was in when last shut down.

[rsjudka]

on the page cycle... it only switches multiple times intermittently right? Just did some more testing and only see it occur every now and then. Thanks for pointing it out, will get a fix in for that.

on the day/night toggle... at that point i feel like it wouldn't really be considered a shortcut, but I can see where an interface for controlling the state of the app would be useful. In the meantime though, i can add individual shortcuts for day/night mode and then buildup that interface alongside it.

[rhysmorgan134]

on the page cycle... it only switches multiple times intermittently right? Just did some more testing and only see it occur every now and then. Thanks for pointing it out, will get a fix in for that.

For me it is everytime, but the way it reacts would depend on the physical characteristics of the physical push button. Its not something that would appear from driving the input via software.

on the day/night toggle... at that point i feel like it wouldn't really be considered a shortcut, but I can see where an interface for controlling the state of the app would be useful. In the meantime though, i can add individual shortcuts for day/night mode and then buildup that interface alongside it.

Yeah I get what you are saying and it makes sense.

[rsjudka]

just pushed some changes up for both the requests. I ended up doing a "renounce" of around 300ms since that's around the average time it takes for a button press (when testing with my button 10ms wasn't enough)

let me know how that works for you!

[rhysmorgan134]

@rsjudka tested the toggle and its working perfect!

Thankyou.

I do have one last point in the key-bindings, I suspect it's not easily possible, as open auto also doesn't support it. But wondering whether it would be possible to capture the skip, play pause etc even when android auto doesn't have focus?

Apart from that I think this features amazing, and thanks for all the work you have done!

[rsjudka]

Yeah i did try that out initially (and realized it wouldn't work). I think my biggest concern would be how would the shortcuts interact if being intercepted by the main window (which would always be in focus).

I'll run a few tests and see if there's some sort of condition i could throw in (maybe only intercept the shortcut if openauto is not visible?) and see what i come up with!

[rsjudka]

alright so it wasnt as easy as i originally thought, turns out you cant just send events to different widgets :p

but with a few tweaks to openauto i was basically able to "backdoor" the keyevents, so go ahead and try it out! (make sure you update the submodule for openauto)

[robert5974]

Impressive👊

[rhysmorgan134]

@rsjudka built the latest version, everything seems to be working flawlessly 😁

I can't think of anything extra for this feature now!