Integrate with HeadUnit Reloaded

[Laurens0619]

Hi,

Thank you for the great work with this product :) I was wondering if you ever considered merging/integration this code with HeadUnit Reloaded (https://forum.xda-developers.com/general/paid-software/android-4-1-headunit-reloaded-android-t3432348)

For me the main purpose of AndroidAutoIDrive is to provide Google Maps on my nbt evo Id4. Google Maps is running through your app but it is currently limited to only viewing the map (also makes sense reading the google maps api)

The most basic integration would be to just start HeadUnit Reloaded, take a screen capture and send it to the car (similar to https://github.com/hufman/IDriveConnectKitDemos/blob/master/screenmirroring/src/main/java/me/hufman/idriveconnectkitdemos/screenmirroring/ScreenMirroringThread.kt ?)

Before I go further, Im curious what your thought and experience on this is :)

[hufman]

Thank you for the suggestion! I have only briefly thought about emulating an Android Auto Headunit for the purposes of capturing the remote frame buffer, but felt it would be too big of a project to handle on my own :) Outright including an emulator's source code feels inelegant, and the phone might melt under the workload of Encoding AA Video Stream -> Decoding AA Video Stream -> Screenshotting virtual display -> Encoding to JPG. Your idea of screenshotting the HeadUnit Reloaded app is a clever way to reuse their work, but I'm not sure how to launch their app within my virtual display. If I could figure that out, I was considering directly launching Android Auto compatible apps instead of running the full Android Auto display. I did test out full screen mirroring, but it prompts the user each time the app starts capture, it requires that the screen stay on, and sending input to the foreground app is difficult to do.

I think my current approach will be trying to figure out how to launch apps on my virtual display, which should provide the best experience that can be expected for such a terrible hack :). I suspect there will need to be some trickery to get around the system privilege check for that, and only be available on rooted phones perhaps.

Of course, I may be missing some other way of doing it successfully! Please let me know your ideas! I have been focusing on the other components more, since the performance of image streaming over Bluetooth is not pleasing to me, so I apologize for the poor shape of the code in that area.

[Laurens0619]

Thank you for the links and information. Running the Android Auto apps in your app would be the best but would be too complex for my development skills.

The workload is indeed far from ideal, however my plan was to run this on multiple devices(phones)

1. One device dedicated running to host Android Auto & feed video 2 connected apps (trunk)
2. The actual (personal) device which will cast to Android Auto

I did a quick test between 2 android devices and Android Auto was working great with HeadUnit Reloaded.

However, I think the screenshotting/encoding to jpg is indeed heavy. I read that GoPro use to have a ConnectedDrive app where you could live stream the camera. Could be of course that they were also snapshotting this but maybe there is another API call which accepts a video stream? Have you investigated this? If not, could be useful if we can find the GoPro app with this videos stream functionality

[hufman]

Oh I hadn't thought of that! That would definitely alleviate some of the processing from the main phone, by running a secondary phone. That does get close to the existing Android Auto kits that are available, however. This project can't be as nicely integrated as a hardware mod, by being limited in the inputs it receives from the car: It only seems to receive button and list selection events, and does not receive more advanced inputs from the Idrive controller or any touch screen events.

The Go Pro app might indeed have some hints! I'll try to dig up an old version of the app, hopefully it still has the BMW resources in it. I think it was only released for iPhone, which is beyond my reverse engineering capabilities, but the resource files might still have some hints. I suspect it's just the regular Image widget, and they only supported versions of the car that connected directly via USB.

[hufman]

It seems that my old iOS6 device is unable to download an old version of GoPro, and the latest version does not include the BMW resources.

[Laurens0619]

I managed to find an old iphone where i was able to load GoPro 2.12.1. No clue if this has the ConnectedDrive functionality as all my data enabled 30-pin cables are either lost/broken. Will try to get a hold of one to test in my car.

I also read that the BMW Lap timer app (also iPhone) has the GoPro functionality. However, what i have seen on youtube so far it seems like this more likely to use screen capture instead of video stream.

Will keep you posted :)

[hufman]

If you are able to jailbreak your phone and extract the unencrypted IPAs, I'd love a copy to dig around in there for the BMW resources!

[Laurens0619]

Made some steps :) First tried if this app was compataible with bmw connected drive. However, BMW connecteddrive needs ios11 to run for which my iPhone 4 is too old.

However, I was able to transfer the ipa from the iPhone 4 and restore it on a newer iPhone with ios 13. This resulted in success and the GoPro app appeared in the BMW Connecteddrive menu.

The only thing, I dont have a GoPro... So I am asking friends if someone has an old which I can borrow and otherwise it doesnt seem too difficult to fake a gopro camera using a raspberry pi with an accesspoint and webserver.

I will keep you updated, if you would like to have a copy of the ipa please let me know.

[hufman]

Oh wow, very cool! I'd love a copy of the IPA, decrypted if possible! That looks so cool, thanks for trying it out! If you extract the IPA, you'll find some BMWAppKitIdentities with cert, and I think there should also be some zip files for the images and texts resources. I believe the actual widget layout (ui_description.xml in Android apps) is stored inside the executable directly, in the one example I've seen, so it needs to be decrypted to get that file.

[Laurens0619]

We were able to emulate a GoPro camera so the app would startup, that is for the latest GoPro app however. The old one we cannot get working.

Decided to put the GoPro project in the fridge until I have a real gopro to test this with. I will continue to investigate how I can get Google Maps App/Android Auto to the BMW :)

[Laurens0619]

I tried your suggestion to use the ScreenMirror test app from your repository. Unfortunately it doesn't work anymore. It keeps searching for Spotify and using an older version of Spotify I am getting a certificate expired/broken error.

I am now trying to build the screen mirror code into your current app but my coding is still a bit rusty so far. If you by any chance have an updated working screenmirror application, please let me know.

[hufman]

The ScreenMirror test app was built with the restriction that it couldn't include any of the car app resources in the app itself, because I'm not sure about the legality of them. Recent versions of Spotify no longer provide the ui_description.xml file, however the cert should still work. I'll try taking a peek when I can. As a workaround, you could include the CarAppAssetManager from this repo and drop the resources into the right place.

[Laurens0619]

Hi Walter,

Thank you for all the directions :) Took some time to figure it all out but we (a good friend of mine) managed to get the screen cap running on the device. Very naste hacked code but the purpose was to have a prototype to check if the ideas was to any use.

As a test I have installed the Android Auto interface on the phone which i mirrored. In the video I control Android auto directly from the phone but the next step would be to use the input from the car controller.

Do you have any ideas which input would be usable? If im not mistaken the bmw touchpad cannot be used right? Alternatively the scroll and navigational directions could be useful.

latency test: (I read from your comments 10fps should be visible so I think we could do more) https://photos.app.goo.gl/P2vGSZQtfKnivrSH7

Google maps/Spotify via Android Auto: https://photos.app.goo.gl/3pesvZufKXsrdJ8q6

Thank you very much for all your support so far. So far it was a very interesting experience :)

[hufman]

Whoa that's great! I also have some good news to share, which might be helpful. I took some time to refactor the map module a lot, to hopefully be friendlier to general image mirroring like this effort. I haven't decided if it's fully generic enough to be merged yet, but it's available at https://github.com/hufman/AndroidAutoIdrive/tree/refactor_maps I also had some partial success with my efforts at mirroring individual apps, which can be found at https://github.com/hufman/AndroidAutoIdrive/compare/refactor_maps...androidauto_apps It requires a rooted phone running the Xposed framework, but Android Auto Google Maps does run and appears functional. Waze doesn't seem to load the toolbar at the top, and CarStream crashes when it tries to load the audiomanager service. Most relevant to you, though, it shows how I map the scroll events and the idrive click event to KeyEvents (tab, shift-tab, and enter) that I can send to the app. I'm not sure, off the top of my head, how you'd send them to the mirrored app, but it's surely possible. The ScreenMirroring demo I mentioned has a branch that uses an Accessibility Service, but Google doesn't allow apps to use that anymore.

Please try out the refactored code (there's further refinement in the second branch too, which I might pull into the main refactor) and let me know any places where I could make it more general. Feel free to reach out directly for faster back-and-forth discussion too, this is exciting :)

[hufman]

For this attempt, I set it to create a duplicate icon from the Maps module, but it should have added some app icons in the Connected menu showing detected Android Auto apps. These app icons should show up just through PackageManager discovery, and don't test out any of the Android Auto connection code until you click them to try to start them. In this picture I have the Maps module disabled so it is just showing the Projection module and detected apps:

I'm loathe to support it officially, because of the difficulty in interfacing with an obfuscated API that is free to change all of its function signatures between versions :) Also, the performance feels really bad, it feels like there's a full second of delay between input and the screen updating, even over USB.

For the input for these Android Auto apps, the connection from the Android Auto host app to the projection app (like GMaps) has an API method for Android Auto to send a KeyEvent to the app directly. I am using the Xposed framework to add some hooks to bypass some authorization checks: The Android Auto SDK running in these apps verifies that the hosting app is signed by Google and has the CAPTURE_VIDEO_OUTPUT permission before actually displaying the app. Android Auto has protocol support for a rotary controller, and GMaps and Waze show a nice highlight to show the active button that would be clicked.

As for BMW, I haven't seen any app examples of touch input or any more advanced input, like how the car's built-in Map can use the tilt of the knob to pan around the map, but I'd look forward to reverse engineering any examples that show up!

[hufman]

Hello, I recently discovered how to use iOS app certificates, so I'm wondering if you were able to extract the GoPro IPA or any other IPA files for old BMW apps? I'd like to have a complete collection of all the known apps :)