search

BimmerGestalt / AAIdrive

Implementations of some Android Auto features as unofficial IDrive apps
MIT License
541 stars 90 forks source link

Google Assistant for Voice Control #18

Closed kohner86 closed 4 years ago

kohner86 commented 5 years ago

I've been experimenting with adding a way for the user to trigger the google assistant voice prompt: https://github.com/hufman/AndroidAutoIdrive/compare/master...kohner86:master#diff-1c29cab35fd57bfdea68ca5f0a671c98

The above largely works but the UX seems a little wonky so I was thinking of putting it in its own 'App' within IDrive (still trying to figure out that part of the API).

hufman commented 5 years ago

Ah, that's a clever way to hide arbitrary extra actions, I like it! Making a new separate App in IDrive is tricky, because the location and icon are hardcoded by what certificate is used to log in to the car. You can see the existing resources I'm using inside the assets/carapplications directory. You can find other certificates inside other BMW Connected Apps around on apkpure and ipahub, but your custom app will need to use whatever icon (and if you enable it, text strings) that comes with that app. If you just want to trigger an event with just a single button press, the am_registerApp might be an interesting path forward: It lets you create any arbitrary menu entry with custom icon (48px square PNG) and text label in the Music or Connected Apps list and register a callback for a button press. It's meant to be a placeholder icon to trigger a full application to load and replace the placeholder icon, but AVContextHandler shows an example of abusing that feature for other purposes :)

I see that some cars mention a speech2text capability, but my current car doesn't implement it so I can't experiment with it yet. The car has some sort of passthrough in use for the Apple Siri feature, but I haven't yet explored how to trigger the same input screen. There are references to iSpeechSupport in the resource files, but I don't see any code in the official apps that touches it.

hufman commented 5 years ago

Here's some info on triggering Google Assistant actions programmatically: https://developers.perfectomobile.com/display/TT/Interaction+with+Google+Assistant+without+voice

hufman commented 4 years ago

image Initial testing seems hopeful, I don't know if it's using the phone microphone or the car's microphone though.

hufman commented 4 years ago

Ah, there is an option in the Google Assistant itself to enable Bluetooth recording! Under Hands-Free, the app requires "Allow Bluetooth requests with the device locked" and works better with "Bluetooth audio recording". image

ziggythehamster commented 4 years ago

On my USDM car which I've coded to have Speech2Text, the only discernable change is that SMS gives me a reply option. If I use that option, it asks me to subscribe to Dragon Drive, which I'm not confident will work so haven't.

This is not the same UI as nav > POI > online search > voice

hufman commented 4 years ago

I haven't figured out what the speech2text feature means specifically.. I wonder if it's related to the Input screen, which on IDrive5+ has an extra entry in the rotary wheel to prompt to record a voice clip and convert the speech to the inputted text. I haven't experimented with coding at all, however.

hufman commented 4 years ago

Any detected assistants (responding to the Intent.VOICE_COMMAND Intent) will be added to the ConnectedDrive menu in the car. My Pixel2 (with Android 9) can reliably trigger the assistant, but my Galaxy S9 (running Android 10, perhaps subject to these rules) can only trigger it while the AndroidAutoIdrive app is visible on screen. I have a wrench icon to (try to) link to the Google Assistant Voice Preferences, which is helpful to enable Bluetooth Audio Recording and to Allow Requests With Device Locked. Only Google currently supports recording from Bluetooth, Alexa and Bixby and Cortana do not. When the app is set to record from Bluetooth, it nicely switches the car to show that it's recording External Input, otherwise the car just shows "Starting App" even though the app is working. Also, only Google and Bixby currently seem to make a prompting noise to indicate that they are listening.

Can I request that some people try out this dev build and see how it works for them?

hufman commented 4 years ago

Please try out this new build, which includes a workaround for Android 10 devices. My Galaxy S9 seems to work very nicely with it.