Screen mirror - app does not show up in bmw & question about waze

[Philip2809]

Screen Mirroring log 2d49d1a1ed48.txt Hey!

First of I want to thank you for this awesome project, I LOVE it, this is just what I was looking for. I have some questions and stuff about this project and if you are free to answer some questions I would love to write a bigger question post to get some more info about AAIdrive, because if I find some time I might work on some features and stuff.

Anyways, I am wondering how I can get screen mirroring to work, I have the addon and when I grant screen cast permission it says "Open the app in the car", but no app for screen cast under BMW Apps appear, I only have the ones that was added initially. I tried starting the screen mirroring first and then start the connection, that didnt help. I have added logs for Screen mirror, it says socket failed?

About waze, I read somewhere you said that only certain package names can access it and send commands etc, with a rooted phone, could this be bypassed? Is this anything you have looked into or not?

I am wondering if you have put up some kind of development environment or do you just always test the code directly in the car? And for the reverse engineering, do you have some base app that you have looked into to figure out what you can send/do? About this, this small info-display, is this anything that can be controlled by BMW apps?

Again, thank you for this awesome project. If you have some time, I would love to ask some more questions, I hope this was not too much :)

For context: I have Android 14, VoltageOS, BMW 218i 2016/2015? MV-130.016.001 (Tried to install new update but was not able to, will try another usb maybe later)

Best regards Philip

Edit: Added link to log as I forgot to upload first, after some investigation I think I it's the init function of CarApp.kt in screen_mirror 2: Just found; https://bimmergestalt.github.io/BMWConnectedAnalysis/ I have some reading to do, very nice to see that you have already written all of this! I totally missed these pages before, sorry..

[hufman]

Good morning! The exploration step is fun, isn't it :) Your log saying "Connected Refused" is interesting, the car connection announcement comes from AAIdrive and it contains the same hostname and port that AAIdrive itself is connecting to (feel free to look in AAIdrive's logs to confirm what it's using). Does VoltageOS include a firewall of some sort? You don't need to start Screen Mirroring or grant permission first, AAIdrive will start it automatically and it should add itself to the car with a new unlabeled circle icon in the Apps menu (and also a labelled Screen Mirroring icon under Media, but IDrive4 doesn't perfectly support clicking those icons). Once in the car app, if you hadn't granted permission yet, it'll show a message asking you to open the phone app and grant permission.

On a rooted phone with something like Xposed anything can be done with enough work! I have an old branch of code where I managed to get Waze to accept the control connection and output buffer from a hosting simulator. Waze worked well because it drew most of its UI, most of Android Auto is drawn natively by Android Auto itself and I didn't want to redraw all that for an experiment. I would like to extract it out to an Addon at some point, but it was very flaky and the hooked functions were obfuscated and subject to changing with any version upgrade.

I typically would test in the car. I did have a headless app that I could watch the logs and confirm initial connections, and recently I started building a fully graphical headunit simulator but it is still pretty basic. And indeed, you found my resources for how I initially reverse engineered the official car apps :) I haven't found any way to use the HUD beyond the music playlist functionality in IDrive5/6. Of course you can ask more questions, they are fun :)

[Philip2809]

Hey!

Thank you for answering, very nice to see that you are still active on this project!

So, I build the screen mirror app myself and got it working, as you say, it was probably the firewall that I accidentally denied it connection to, because it does have a firewall. About the icons, why are some unlabeled? Is this some limitation? Also with the detailed car info, why does the navigation work the way it does?

Once I got screen mirror working, I got an app to change my phones resolution to better fit on the car display and found out that if I screen mirror a certain app and then use something like this: https://play.google.com/store/apps/details?id=com.ss.screenoffandplay&hl=en to turn off the display while letting the app run, I can keep the phone "off" while screen mirroring. I have denied all these extra apps network permissions and I have an adblocker so I don't care about the ads really but if this works well I might build a addon that does this for AAIdrive.

When using the android auto waze, was the performance any better or is it still limited by the speed of the usb/bt? How come that android auto on other cars can run with more fps and this is limited, is it because it is pictures etc? Because if its not any better I don't see a reason to not just use a screen mirror.

My idea is to build a carapp for controlling waze, is the "settings menu" when moving the knob to the right editable? Because my idea would be to add "Report accident" or "Still there" etc. buttons in a menu and then make a "clicker" to execute the user request.

However before I can do alot I need to figure out why my AAIdrive keeps crashing, I have disabled battery optimization, what else could cause problems? I have a ram monitor that I try to keep open and it runs around 68-78% and crashes even then.

I have IDrive4 from my understanding and when using the scroll wheel it says "not available" or something, however once when I used crDroid (another OS) I got information about the playlist, however my bt sound was bugged so I changed to voltage.

If I want to check on the official apps, is it just to download like Twitter and watch the bt connections?

Thanks again for your help!

[hufman]

The car app resources are cryptographically signed by BMW, including the entry button icons and their labels. That circle icon is provided by the Samsung SmartThings app, and while I can't change the icon or text, I can hide them, which I do to try to reduce confusion. The car info navigation is in line with the rest of the car, where a text box opens a submenu of options. What about it seems questionable?

I'd be interested to see what you find, perhaps with information about how the screen off feature works. It would seem easy enough to integrate it into the existing addon, if possible. How does touching the screen work while it's off?

The Waze hack wasn't any faster, due to limitations of the protocol. The Android Auto protocol sends a video stream to the car over USB or WiFi Direct. BMW apps talk over USB or Bluetooth, and can only send BMP/JPG/PNG pictures to widgets in the car. It is much faster over USB than Bluetooth, but still dependent on the size of the image (including complexity increasing the data size of the JPG).

The advantages I was hoping to achieve with native AA app hosting were to gain control to send input events to the app and to run the app without taking over the phone screen. And now, running Android Auto on the phone for Screen Mirroring is less convenient, since Google took the phone interface away again. I suppose it might be possible to recreate the Dev Headunit as an addon, perhaps, tho not very convenient.

The side bar on the right side of the screen is not editable from a car app.

Are you using the new beta from the Play Store? Someone recently reported an Issue with it seemingly crashing after a few minutes, and I would appreciate if you could find logcat logs showing the problem, since it works for me.

The car natively talks AVRCP 1.4 and can view the current queue over Bluetooth if your phone allows it, and can also browse some apps in the Bluetooth screen. However, the hud does not show any playlist from any car app before IDrive5.

Your car doesn't talk the apps protocol over Bluetooth, you'd instead want to tcpdump the localhost connections on your phone to monitor the traffic from the apps to the proxy into the car. If you do find s car with Bluetooth Apps, you could hcisnoop the traffic, but my Wireshark plugin is buggy and abandoned since tcpdumping was better.

[Philip2809]

In the car info navigation it opens a list with opens which i can click, if I click it I get the info with a button at the top and to get back to the list I have to select which can info I want to see, however clicking on any of the other text in the detailed info, it opens a page without any data. Can attach some pictures soon if you want.

That is both sad and weird that they require the buttons to be signed, but I understand now why it looks like it does. I tried the screen off app and waze was a little buggy? The car position did not update but the speedometer did, will need to look into that more, you cant use the touch screen yourself because it uses that to detect when you want to stop using the screen off thing. If the right side bar is not editable, how come the audioplayer has a "Random" button for its sidebar, or is that some default BMW stuff on the mediaplayer?

I did manage to get some logs: AAIdrive log 7fe6a61229d0.txt AAIdrive log e0b62830d1bb.txt AAIdrive log 2f2f3fbbd6df.txt AAIdrive log 214db9959faf.txt

But I can't really figure out exactly what is doing the crashing. Before I used the app from the play store, non beta, but recently I have built the app myself to try sentry but I do not get any crash reports there.

The car does not support bluetooth apps, so I will try to debug my tcp connection sometime. I am guessing everything is unencrypted with the usb connection as well?

Edit: here I think I have found a perfect, got this right after the crash AAIdrive log 5df0b2dc3e8b.txt

[Philip2809]

I know this is more about AAIdrive, should I write there? How exactly can I add a label on for example the music page? Or is the layout predefined and I cant change it? How does this work? Trying to understand the code, I guess the "fits" function has something to do with this?

[hufman]

So, while the third-party apps (such as Samsung and Spotify) have flexible menus, and the app can add extra handling and decide what screen to open after clicking a button or list item, the Car Info app uses a built-in BMW app (News), and these buttons are hardcoded to open specific screens and can't be changed.

What Random button do you see? Do you mean the left toolbar? The toolbar is just a list of buttons, and each button has an image from an icon pack and a string label, so AAIdrive can reuse the shuffle icon and set the label according to the state. Do you mean the sidebar that can be toggled open or closed and shows navigation instructions or the current playing music? That is the sidebar I refer to, and it is on the right side of the screen in my car.

I merged the branch so the new API34 code is the main branch, and so now the Github Latest Build has it now. Feel free to run this version for a while and I'll watch for any emails to arrive in my Sentry account. Since you have the Screen Mirroring addon, can you check something? When AAIdrive closes, all of its apps will be removed from the car, but any addons are managed separately and may keep running. If AAIdrive shuts down but Screen Mirroring (or other full addons like Hass Gestalt or ReadYou) keep running, then indeed AAIdrive is at fault. If everything shuts down, then the BMW connection itself has closed, and it's harder to test. MyBMW is flaky with USB, I'm surprised you have it working at all!

Since you've set up the project to build, you can look at what the car app resources look like: In your project directory, look in app/src/main/assets/carapplications after Gradle has extracted them from the external APKs. You ask about the audioplayer, so open up the multimedia/rhmi/ui_description.xml file to see the menu layout that is uploaded to the car. Instead of hardcoding the resource IDs, I built the fits function to locate what state is suitable for which needed purpose in AAIdrive, and the music app logs which ones it found. It almost certainly used toolbarHmiState id 16 for the main screen, so scroll down to line 961 to find its definition. You'll see all of the components defined here, and what data models they display, and any special properties they come with. The only thing that can be changed by a car app are the component properties and the data models. For example, the Screen Mirroring addon moves the cover art image and changes its size, by editing these properties. Adding new components would require editing this file, which would require breaking the RSA4096+SHA1 signature on the BMW certificate. This ui_description also defines the entrybutton component, which is where the app icon and label are defined, except changes to their models don't take effect after the app is loaded.

[Philip2809]

Hey, sorry for a late response, have been up to a lot lately.

What I meant by "settings/options menu" and the random button was this: I get this when I move the knob to the right. I now understand that this is not changeable and the random button is something that the multimedia view always has.

About the playlist thing, I have tried all AVRCP versions in developer settings yet this is what I get: I have also tried different music apps, will try to look into the bluetooth data being sent to maybe figure out the problem.

About the crashes, sadly when it crashes it crashes fully, all apps gone. I did a tcpdump and got this: bmw_traffic.pcap.txt (Remove .txt added for upload) However in this dump it connected, crashed then reconnected and it's over 1700 records, so for me it's a needle in a haystack, don't really understand it yet. What's the difference between "ETCH" and "TCP" in this case? Also what does port 51464 do?

If you want I can use your build but I tried sentry but did not get any crash reports at all. Right now I am playing around with the menu options and stuff, also I can only see the song title and band, isn't the album supposed to be shown as well? And my last question, is labels a item that can be selectable?

Again, thank you for helping me! I am glad I got this working, even if it's sometimes crashes, I would love to look into how, if even possible I can improve it!

[hufman]

Ah, I haven't used IDrive4 in a while and perhaps have forgotten some of the version differences, apologies. In the Wireshark interface, you'll see each discrete packet as a separate row. Each packet will display the highest-level protocol that was parsed out of it, such as TCP or Etch. Packets that just show TCP don't contain any Etch data, and are just acknowledgement packets, while the Etch packets have the actual application-level data. The BMW app hosts a server at 4004 and proxies any incoming data to the car over USB. Port 51464 is just an ephemeral client port, each AAIDrive module establishes its own connection to the car. The Etch protocol compiles function and parameter names down to hashed identifiers in the actual protocol. You'll want a copy of this file, which you can give to Wireshark and get human-readable names from the Etch packets.

When you say "crashing", does it take the AAIdrive phone interface down too? If there aren't any crashes in the logs or in Sentry, then it may not be AAIdrive crashing but perhaps MyBMW or BMW Connected. Since you say everything disappears, including Screen Mirror, then it would appear to be the MyBMW connection to the car that is disconnecting.

IDrive4's multimedia sidebar for global metadata only has two text fields, which AAIdrive fills with artist and title. IDrive5 has a separate system for showing media information which includes cover art and track time or album information. I'm not sure if labels are clickable, the ui_description has an action attribute on any component that is clickable.

[Philip2809]

I see, will try the file for Wireshark as soon as possible and will enable the debug logs for the MyBMW app as you state in the analysis documentation, to try to figure out if that maybe gives more information. I am not sure if the AAIdrive app crashes, but I don't think so, I think it's rather the MyBMW that crashes, but will look into it!

When you say "sidebar" what exactly do you mean? Now when I connect my phone with only bluetooth to the car I get the track time, track name, artist and album name. The track number is always 1/1. However if I use an iPhone the same information is there but the tracks position in its album is also there. I will upload a picture of this later or find some online later, also realized I can try to find the screen I have seen once before online. Will look into it and update here.

Also, I have got a accessibility service up to interact with waze to automatically report anything that happens, will now try to figure out to make a iDrive screen for interacting with it!

[hufman]

As I remember, IDrive4 has a slide-out sidebar on the right that you can toggle on and off, and shows selected information no matter where you are in the system. One of the options to show here is Multimedia Info, and car apps like AAIDrive can set information here. That is what I refer to as a side bar. As for regular Bluetooth, I know BMW has much weaker support for Android, and in fact IDrive6 still doesn't show cover art over the Bluetooth connection from Android. I don't know about any other metadata, though. I have an old branch of the Screen Mirroring code that uses an accessibility service to interact with the mirrored screen, if you'd like to try it. I wasn't satisfied with the performance and the heuristics to determine the order of scrolling through the selectable widgets, but perhaps you might find it useful.

[Philip2809]

I see, my IDrive is not wide screen, its smaller, only a single page at a time, like this: I managed to get the latest update of IDrive I can have, version: TV 130.025.041 & HV 130.025.041. This also added an "Add-on version", do you know what this is?

Something alike what you already have done was my next idea to do but it seams you have already thought of everything :D

I also get this in my multimeda view; however when clicking "Spotify" or "Screen Mirroring" here I get to the "Detailed car info" page instead of the correct page. Any idea why this could be happening? Will try to look into the tcpdump, but here they are; these also include a crash at the end. I havn't managed to get the lua plugin working yet as I used wireshark 4, which version would you recommend? bmw_traffic3.pcap.txt bmw_traffic4.pcap.txt

My "Audioplayer" looks like this; Is this on purpose or this happens automatically as its a smaller screen? I see that in your gallery you can see the album easily, is it hidden "below" the other texts or just not drawn at all?

Using the accessibility service I saw the same thing you saw the the order of the items but will look into your code as well!

[hufman]

You don't want the Lua plugin, that is just to add support for decoding the btsnoop packets into Etch packets. You only need the ewh file to decode the Etch symbol names. The Spotify (and other music app) icons, the Phone Notifications icon, the Screen Mirroring icon, and the Detailed Car Information are a special type of icon that isn't full-featured, the car protocol uses them as placeholder icons before launching the full car app with the signed resources restrictions. With IDrive5/6, the event handlers for these icons are allowed to open other RHMI apps and the process works well. For some reason, IDrive4 does not, and the commands to show the relevant app instead open a different app. I don't know of any workaround for this.

Ah, I see what you mean now, this is the compact Audioplayer interface. The screen technically has both sets of widgets, but conditionally hides them based on whether the screen is in widescreen mode or not. I don't know of any workaround for this either, I'm afraid.

[Philip2809]

I can't figure out how to import the etch decoder to wireshark, can you help me?

That's kinda sad that the icons buttons does not work for IDrive 4 :/ Anyway I am trying to understand how the ui_desc works now better and I have some questions. When making lists, how does that work does it need one of each element? For example in EnqueuedView.kt it makes sure that there is at least one list, one image that does not have a position x (why, if this is editable?) and it has separator(s), am I understanding this right? Is the multimedia ui_description a default bmw app? If I would want, I could just move around the text and images to get the view I want right, because if I understood correctly, properties etc are fully editable only the amount of labels, images, etc. is fixed?

And when it comes to stuff like this, I really like to visualize to understand what is going on, so I made a little script to show the xml, I used the audioplayer in this example. Green is images, red is text and blue is the gauge (put a default height of 50 for this element as it didn't have it defined). With conditionValue = 0/2: The rest of the elements are off to the side. With conditionValue = 1/3 And there the rest of the elements are off to the side. You noted in your analysis that it appear that the conditionValue was 1 or 3 when the sidebar was open, which is the same scenario for me, but instead of more screen space my screen is just smaller, so it's always at that value. Adding this to this post might be stupid but I just wanted to say this helped me a lot to understand what was going on, I tought that the position of elements was added/modified by AAIDrive so I was confused when I found no code showing this.

Edit: Forgot to add this; Could you explain how the LIST_COLUMNWIDTH works? 0,100,*? Edit 2: Looking around a bit more in the ui_desc I found instrumentCluster at line 551 in the mediaplayer, this references the playlistmodel 534. How could I try to send some data into this model? Edit 3: Do you know how HMISTATE_TABLELAYOUTand HMISTATE_TABLETYPE works?

[hufman]

In the Edit > Preferences window, navigate to Protocols > Etch, and set the Apache Etch symbol folder to the location where you've saved the ewh file. Then you'll start getting symbols in the protocol field of the main interface:

That's a really cool visualization!

I'm not sure what you mean about "when making lists". The fits functions in AAIdrive are helper functions to locate an HMI state from the ui_description that has the right widgets for the specific view, and it just needs to differentiate from any other states in the file. For example, the EnqueuedView checks that there aren't any Separators because the BrowseView uses the pages with Separators. I'm guessing I added the check for an image without an X position because it later finds that same element to use as the playlist cover art, but really no other plain HmiState has any images. If an element doesn't have an X/Y position specified, the car provides a default flow layout, similar to how HTML is laid out without CSS positioning. You are welcome to change the properties to move widgets around, sure. I believe it would break the LayoutBag conditions, but they don't have an effect for you and so that would be fine. Indeed, AAIdrive doesn't have any code for the LayoutBag conditions, the car handles that entirely on its own. In fact, AAIdrive has no way to tell if the widescreen sidebar is open or closed, which is why the Map has the setting to specify widescreen mode.

Audioplayer is the official music app from BMW Connected Classic, and a very similar layout is used for the 3rd party music apps in BMW Connected (Spotify, iHeartRadio, Pandora)

LIST_COLUMNWIDTH is pretty simple, it's a comma-separated list of pixel widths for each column with an asterisk to make the last column expand to the remainder. HMI_TABLELAYOUT and HMI_TABLETYPE are less understood, I think it's something to do with a screen layout template in the car. I use it in the Notification app to make the notification list expand to the full width of the screen, but the same setting on the EnqueuedView doesn't seem to have the same effect. The instrumentCluster model is populated here, based on how I observed Spotify to behave.

[Philip2809]

Thank you, got my Wireshark working, will try to see if I can get any info out of it!

I am starting to understand the system now, but I am also starting to have more and more questions, if you don't mind.

When it comes to the Car app, each Car app can only have one ui_desc each? I can't use a view from both SmartThings and Media Player in the same App? Can two views use the same state as it is with the view files today as long as they are not opened from each other? Or is it strictly once per app? Weird question maybe but why didn't you use the ids in the ui_desc files, why make the fits function? Wouln't it be easier to say use state with id of x? Also have you compiled a list of what states is available and their components etc? If not I would like to do this to get an overview over what I have available. Also can you get the current property value of an item? If so that could be used on certain states to read if some elements, in the case of the mediaplayer to check if the album image is off to the side or not.

When it comes to the models, which ones can have a "value" that can be shown in the UI? For example, imageIdModel can only have imageId so this can't be used by us? Only raImageModel can be edited by AAIdrive?

Where do you hide the app name and icon in the connecteddrive list?

When i mean the list I was wondering if you needed image components to be able to show data in them, but based on this I guess I don't need that?

val appsListAdapter = object: RHMIModel.RaListModel.RHMIListAdapter<MusicAppInfo>(3, apps) {
        override fun convertRow(index: Int, item: MusicAppInfo): Array<Any> {
            val appIcon = graphicsHelpers.compress(item.icon, 48, 48)
            val checkbox = BMWRemoting.RHMIResourceIdentifier(BMWRemoting.RHMIResourceType.IMAGEID, musicImageIDs.CHECKMARK)
            return arrayOf(
                if (item == avContext.controller.currentAppInfo) checkbox else "",
                BMWRemoting.RHMIResourceData(BMWRemoting.RHMIResourceType.IMAGEDATA, appIcon),
                item.name
            )
        }
    }

I will try to look into the instrumentCluster but that is maybe not what I am looking for. Because it does not do anything in my instrumentCluster. What you say about the positioning is very interesting, is it based on the order of the items in the components? Do you know if we can by properties set LayoutBag conditions?

Edit: forgot to add; can you change color of text? Edit2: Why do musicplayer use mediaplayer and not spotify? Spotify has a lot more images, it is because of mediaplayer has better states and my question before is like that you can only have one per app? If that is the case, can you use the pictures anyway by adding them as images and not imagedb?

[hufman]

These questions are super fun, I'm really enjoying them :)

As far as I know, each RHMI app can only have a single ui_description, yes. The RPC calls use an app handle ID and a numeric model/component ID from this description, which would make it impossible to have multiple ui_description files loaded for the same app handle. The Views classes in AAIdrive are just an abstraction on top of the states. For example, the Music app instantiates several BrowseViewPages as you browse deeper into the library but reuses the same 2 or 3 states, and the Filter and Search Views reuse the same Input state. The car is merely told, in a button callback, to open a specific numeric state.

When I first started the app, I wasn't sure how different the ui_descriptions would be for different versions of BMW Connected, and so I didn't want to hardcode any IDs. Users provide their own external APKs for the build process, and I didn't want to require specific exact files. The multimedia files have been relatively stable (because the old Spotify/Pandora/iHeartRadio apps needed them to be stable), but the BMW One internal description changes a bit across different versions, and I even had to dynamically find the imageId for one of the icons. I haven't compiled a list, precisely. I did scan through the apps in my collection to find what permissions and capabilities of each (for example, only News provides the readout command). The Multimedia and OnlineServices resources are relatively standardized and flexible, because they are meant for 3rd party apps to use. In fact, BMW Connected internally has names for the states and components, but I have been using them for other purposes so I've ignored the official names. If you'd like to document what you find, feel free to contribute to the documentation :) I haven't had anyone else show such interest in building custom apps, so I've never prioritized documenting what features are in what apps.

There isn't any way to retrieve the model value or property from the car, but the car only knows what is specified in the ui description and any RPC calls. This would not provide access to the LayoutBag's current value, alas. I don't know any way to set any LayoutBag properties, and I just assume the normal setProperty would replace the LayoutBag with a single value.

Technically all of the models are meant to be shown in the car and are meant to be updated from the app. textIdModel and imageIdModels are integers that point to resources in the texts.zip and images.zip files, which limits their use by AAIdrive. For example, the Music PlaybackState changes the imageIds shown in the toolbar, but further alterations are difficult. raDataModel, raImageModel, and raListModel are more free-form. raListModel stores tables of information, and ByteArrays of images can be put in a cell and will be displayed, but the car limits the table row height to 2 text rows and so any images are likewise limited to ~90px. This is why the custom map's preview map is so small. The EntryButton use textIdModel and imageIdModel to specify the icon and label, so they point to a specific label and icon from the resources. To hide these labels, in case they would be misleading, AAIdrive just doesn't sent the texts.zip or images.zip files. A side conversation discovered that it's possible to get past the security system if you can get root access into the information, which is exciting but probably infeasible for most people.

I do see code in the old Connected Classic app that means that the legacy Audioplayer should support setting the InstrumentCluster playlist. Since I don't have an IDrive4 car to confirm, you should try installing BMW Connected Classic and seeing if the official Audioplayer app shows up in your InstrumentCluster. I believe my code is based on what I tcpdump'd Audioplayer doing, but maybe I missed something.

The Richtext models, used by Notifications to view the full notification supports a few formatting things (<info> and <color=RRGGBBAA> and maybe others, per experiments), but no other widgets do.

Spotify uses a newer multimedia ui_description that includes <audioHmiState>, and I believe is only supported by IDrive5+. This audioHmiState would be preferred, because the car controls the positioning and so it integrates better and looks like the other music sources in the car, and it's the only way to set the global coverart. I had lost access to my IDrive4 car by the time I implemented the Spotify resources, so I don't know what would happen if this was sent to the car. Feel free to remove the isId4 checks from here to test :)

To reiterate, I am really enjoying these questions :) Feel free to reach out directly!