[REQUEST] Virtual Cockpit use CarPlay google map or apple map

[Gold-TW]

I wonder if this idea can be implemented?

Virtual cockpit using CarPlay Output google maps or apple maps.

Or can install GOOLE EARTH in MIB2 high

Thanks for the suggestion.

[grajen3]

Some success:

Slight problem that it's really slow to parse currently with my implementation + I don't yet now how to force re-render of displayed audio text (I manually switch views to refresh)

[olli991]

My understanding is the Audi cluster screen is essentially the same as the main screen on other makes, it's connected with a normal monitor connection, eg. DVI or similar, not the MOST connection we're dealing with. As it's connected as a real monitor on the Tegra it can be rendered directly to.

Okay I just checked. TT cluster houses MOST and LVDS. So yes you could be right... Carplay stream might go via LVDS then

[OneB1t]

My understanding is the Audi cluster screen is essentially the same as the main screen on other makes, it's connected with a normal monitor connection, eg. DVI or similar, not the MOST connection we're dealing with. As it's connected as a real monitor on the Tegra it can be rendered directly to.

Okay I just checked. TT cluster houses MOST and LVDS. So yes you could be right... Carplay stream might go via LVDS then

MOST will be there for audio and also for map streaming same way as on MIB2

[grajen3]

Turns out using stream.available() to check if there are any more bytes was not good idea lol - after just readjusting code a bit it's much more speedy - for now (tho I didn't have full 10MB trace produced to test worst case scenario with current implementation as I do read entire file - possibly could skip stuff that I already parsed in previous parse attempts of given file) - reading/parsing took ~180ms and I sleep for 1000ms before trying to re-parse last file - so so far worst delay is ~1.2s + whatever it takes for traceserver to dump stuff to sd + whatever it takes for VC from my forced refresh for it to actually re-render

https://github.com/jilleb/mib2-toolbox/assets/152773682/4140059e-7f43-46e3-a880-d95947a097c3

next steps for me is try to figure out how to interact with VC next turn stuff - as I mentioned - audio meta is least interesting for me to have in VC, but was just easiest to find in LSD. I won't bother with cover art for now. For hacky stuff I guess I could just dump next turn info on audio screen to have some usability of that information being available, but I'd rather do it properly so it does show in middle of my right gauge (like what happens when native route guidance is active)

Once I get next turn somewhat working, I'll start cleaning up my code and have something out in the open - probably pull/merge request against Andrew's repo (I have bunch of mess in my working directory for unrelated stuff, so would need to start from some clean branch heh)

[jilleb]

Wow this is great progress!

@grajen3 , the cluster should be able to handle info about next turn information as well, i've seen things like guidance information and such.

[grajen3]

@grajen3 , the cluster should be able to handle info about next turn information as well, i've seen things like guidance information and such.

Yeah - I do see next turn information from native VW navigation in my right guage (where compass is in my video / native route guidance is not active) - possibly there are other views that render that info as well in VC (maybe even HUD, tho I don't have HUD hah).

So that's what I want to get now, but will likely be trial and error - so far I didn't found where it could be, but I also didn't spend a lot of time on it as I didn't have data to feed into it, so motivation for that wasn't high yet, just few quick searches - but that's next on my list now (also much higher motiviation to find it now :D)

[OneB1t]

yes MIB3 is able to show turn data with this older VC @grajen3 can you please share some code how you achieved this? 🤪

[grajen3]

Heh, sounds like I'll do some cleanup now to share the code so maybe others will join me as well :D

--edit

for next turn I do think it will be more challenging than audio overall, because next turn "UI" is not even shown unless native route guidance is active - so I suspect we will need to find where UI decides what to show to either patch it to allow extra codntions for AA navigation or do something hacky to trick it into thinking that native route guidance is active (I did encounter situations where some files couldn't easily be patched as thinks were falling apart completely)

[grajen3]

@OneB1t https://github.com/grajen3/mib2-lsd-patching/commit/4e302ec72bf55e141eb843824b22b7d4c21a972a#diff-6fd7b601401b91bc06d26b942a350063cce4bf7e817c8bc6a30c30266c7987ff is more or less this stuff - I didn't try if this actually works other than if it builds (I was doing it from clean slate and just grabbing relevant stuff) - I won't have time to test it soon, but at least you can see more or less what the code is - my scanner is pretty verbose and very messy being dumped in one of existing android auto related modules :D I blame java (but more likely my inexperience with it lol)

[grajen3]

https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/navsd/functions/TurnToInfo.java this is likely where I could feed next turn information to VC and https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/navsd/functions/DistanceToNextManeuver.java where distance to that event would be (other files in that dir have other nav related meta). Still will need to find how to convince VC to start showing next turn UI at all (maybe just filling any information in those would enable it, but I doubt it). But I need to take a break for now, maybe will have next hacking session in the evening or tomorrow morning

[jilleb]

Awesome hacking :-) On Seat and Skoda virtual cockpits (and I think VW Polo as well), there's a setting that allows you to change which data is shown on the clocks, with the following dialog:

So there it might be a case of just offering the data to the right channel, selecting the navi guidance setting for one of the clocks, and then it should work. By default it's showing "Mobile navigation active" or something like that, when Android Auto/ Carplay navigation is being used. I forgot how the VW Passat/Golf VC has their clusters set up..

[OneB1t]

for next turn maybe TurnToInfo.class can be place to modify? :-) protected void setTurnToInfoStatusData(TurnToInfo_Status var1) { NavigationTurnToInfo var2 = this.getNavigationService().getTurnToInfo(); var1.turnToInfo.setContent(var2.getTurnToInfoStreet()); var1.signPost.setContent(var2.getTurnToInfoSignPost()); }

[grajen3]

Awesome hacking :-) On Seat and Skoda virtual cockpits (and I think VW Polo as well), there's a setting that allows you to change which data is shown on the clocks, with the following dialog:

So there it might be a case of just offering the data to the right channel, selecting the navi guidance setting for one of the clocks, and then it should work. By default it's showing "Mobile navigation active" or something like that, when Android Auto/ Carplay navigation is being used. I forgot how the VW Passat/Golf VC has their clusters set up..

Mine doesn't have that much customization - it does have 5 or 6 presets (like "assistants" which I currently have that does show next turn in right, but I can't freely select which data is shown in left/right guage. I saw newer VW than mine do have similar option to customize, but mine seems a bit older :)

https://youtu.be/-veJNqGaAAI?t=17 this is what I have (first thing I found that I could reference) - those presets are as-is and there is no further customization - most I can do is turn on/off some data to be shown in center view for "driving data" view (or something like that - one that you can scroll through stuff like speed/consumption/oil temp/trip distance etc)

but assistants view I use do show native navigation next turn info if native route guidance is active - if it's not active it does show compass

[OneB1t]

those options are available for 10" 2nd gen VC which was installed to passat and arteon first generation is bigger 12" but with less settings

3G0920790 vs 3G0920320

[grajen3]

for next turn maybe TurnToInfo.class can be place to modify? :-) protected void setTurnToInfoStatusData(TurnToInfo_Status var1) { NavigationTurnToInfo var2 = this.getNavigationService().getTurnToInfo(); var1.turnToInfo.setContent(var2.getTurnToInfoStreet()); var1.signPost.setContent(var2.getTurnToInfoSignPost()); }

Yeah - seems like we both found these :) ( https://github.com/jilleb/mib2-toolbox/issues/159#issuecomment-1870269969 )

[OneB1t]

maybe just run normal navigation in background + navpatch? then VC will be able to receive those messages (and also modify it to show android auto data if available if not keep internal nav data)

[grajen3]

maybe just run normal navigation in background + navpatch? then VC will be able to receive those messages (and also modify it to show android auto data if available if not keep internal nav data)

I think this is at very least good way to validate that those files are ones that feed information to place we want them to. Starting some native navigation every time seems tiresome. I guess you could have some guidance to destination you never will reach started and I think that navigation is resumed if you turn you car off and on again, so maybe? But it would show a mess in map viewer (at least if you intend to use that instead of some custom dashboard :) ). But this is probably good next step to figure out if those are the ~droids~ classes we look for

Completely aside, but in the past I also toyed with trying to get google maps trip sharing to use that to automatically launch same route in native VW navigation - https://github.com/costastf/locationsharinglib has a way to retrieve that info (there's endpoint you hit that gives ~json info back) -from that you could get actual route that google map figure out for you as list of latitude/longitude coordinates (I did that) - but I had troubles to programatically launch it in LSD - I did find places that you could feed gpx, but it wasn't expecting that information at this point, so there likely had to be other steps executed first + you had to manually share information on each google maps navigation as there is no persistent "always share trip progress with that person" and I didn't really want to try to hack that part in on android side (completely unknown to me) and also there is only one account you can share trip progress in android auto, so you could only share that with VC OR someone you might actually share your trip info with :) So in the end I did scratch that approach

[grajen3]

TurnToInfo was not exactly what I was after :) but it "does something"

after checking TurnToInfo I also realized we aren't really getting "total remaining distance" and "estimated time of arrival" in events I parse, will check if there are other traces to maybe also parse to get those

https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/navsd/functions/ManeuverDescriptor.java is more likely one to use for right hand side and https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/navsd/functions/RGStatus.java suspect for setting route guidance status

[OneB1t]

it looks like GAL actually does not log those values they are probably not even a part of Android Auto sensors data https://milek7.pl/.stuff/galdocs/huig13_cache.html

so this is not going to work

[jilleb]

it looks like GAL actually does not log those values they are probably not even a part of Android Auto sensors data https://milek7.pl/.stuff/galdocs/huig13_cache.html

so this is not going to work

But why then does AA on MIb3 show the data on the VC?

[OneB1t]

maybe added those in next implementation? but if i go trough all logging traces in GAL there is nothing like "arrival time" or "distance" in our version

--edit

maybe this? but it is probably just for next turn [DSIAndroidAuto2Impl] %s : distanceMeters=%i, timeSeconds=%i, valid=%i [NavigationStatusCallbackHandler] %s : distanceMeters=%i, timeSeconds=%i

[grajen3]

maybe added those in next implementation? but if i go trough all logging traces in GAL there is nothing like "arrival time" or "distance" in our version

Yeah, probably - same like actual "proper" instrument cluster display ( https://developer.android.com/training/cars/apps/navigation#drawing_to_the_cluster_display ) that is implemented in recent BMWs and Volvos ( https://youtu.be/BlIOyNvBaeQ?t=63 for how it looks like ) - but this would be waaaay too tricky to do for us :D

Maybe this could be done on adnroid itself to capture info- https://developers.google.com/maps/documentation/navigation/android-sdk/reference/com/google/android/libraries/navigation/Navigator.RemainingTimeOrDistanceChangedListener - there are APIs for this - but this is behind "contact sales" and I won't be touching android / google maps itself to try to reverse engineer acceess to it heh. Possibly could be read from notification itself ( https://github.com/3v1n0/GMapsParser ), but I have no clue how to get data from android to headunit that is something usable (if they are on same wifi network, I guess there could be comms like that)

maybe this? but it is probably just for next turn [DSIAndroidAuto2Impl] %s : distanceMeters=%i, timeSeconds=%i, valid=%i

That's next turn distance :(

[DSIAndroidAuto2Impl] onJob_updateNavigationNextTurnDistance : distanceMeters=6452, timeSeconds=340, valid=1

But honestly - this is much less important to me than next turn information, so worst case scenario if needed I will just try to blank that in UI (my screenshot shown data from native vw navigation that I did have running) - for ETA I'm fine with peeking onto headunit - next turn info is much more valueable to have closer to line-of-sight while driving to not "miss" a turn :)

[OneB1t]

if those devices are on the same network we can just create simple http endpoint on Android device then pull it from MIB device

i been thinking about it to send AA image so i can draw it but problem is how to get screenshot from android auto.. (that is possible iniside developer settings but no idea how to make it automatic)

--edit i think that what is actualy going on on those BMW is that they are running google maps inside cluster itself and then just feeding it data from AA

[grajen3]

i think that what is actualy going on on those BMW is that they are running google maps inside cluster itself and then just feeding it data from AA

No - there is actual google maps android phone second projected "window" for instrument cluster - you can see this in action on your PC with head unit emulator ( https://developer.android.com/training/cars/testing/dhu#cluster-display ) - I've done that for testing and trying to render it on android phone to screenshot it myself :D but failed miserably hehehe

https://developer.android.com/static/images/training/cars/dhu_instrument_cluster.png has nav meta available in more recent android auto "servers" which does have ETA etc

[grajen3]

if those devices are on the same network we can just create simple http endpoint on Android device then pull it from MIB device

Yes, but it's somewhat messy for every day usage - for that I would need to hotspot from my phone and have car connect to it - that would be fine if I would be only driver. The other option of phone connecting to car's wifi is bad for me because of how bad GSM works on the car (maybe that's just my car hah) and phone would constantly lose connection try to fallback to it's own GSM for internet etc. Maybe if I would have dedicated phone to just sit in car for AndroidAuto and hotspot that could work, but I don't have spare one for that and won't be buying new one for that :) I also use AAWireless so that alone is Wifi that my phone is using (but it has no internet so no problems with internet connection switching) - so I would have to give up on that.

There is custom communication channel that you do get via exlap that GAL we have does expose - in fact @jilleb used that in https://github.com/jilleb/mqb-pm (tho to read data from CAR and not to feed data to car) - but exlap generally allow for "function calls" - but this would be whole new thing to dig into to add custom functionality to it

i been thinking about it to send AA image so i can draw it but problem is how to get screenshot from android auto.. (that is possible iniside developer settings but no idea how to make it automatic)

video dumping is "always on", but it stores data that non-rooted phones have no access to (from other apps) for sEcuRitY reasons - you do get access to it when you connect your phone to PC with usb file transfer - you could extract last frame on phone itself if you would have rooted phone, but you are still left with having to get that to the car :) I saw Andrew mentioned there is a way to dump video stream on car itself, but problem is with decoding it to extract frames from it?

[OneB1t]

yes you can dump AA on MIB unit itself but that is H.264 encoded and there is no simple known way to play it on unit

but there is other option as GAL binary is basically decoding AA stream received over USB to show it on screen. Somewhere in the decoder it will be possible to store "current screen" and save it to filesystem (hacking gal along this way (NvSSVideoDecode, NvSSVideoDecode) -> libnvss_video.so -> libnmsdk.so / libnvmedia.so)

unfortunatelly my knowledge about this is still low but it can be done for sure it just require proper knowledge of arm binary dissasembly as it will require quite extensive modification of that code

also it will be interesting to mess with this part of configuration

  # DISPLAYABLE_EXTERNAL_SMARTPHONE = 59 (previously used DISPLAYABLE_EXTERNAL_DVD_VIDEO = 34)
                "displayableID":59,
                # preferred renderer (defaults to lowest available if preferred is not available)
                # 0 - default
                # 1 - nvss
                # 2 - nvmedia
                # 3 - qc
                # 4 - cinemo
                "type":0,
                    "nvss":{
                    # NvSSVideoOutputDevice_LVDS = 0 (default), NvSSVideoOutputDevice_HDMI = 1
                    "outputDevice":1,
                    # Setting provideTimestamps = false disables the NV decoder "late frame dropping" functionality
                    "provideTimestamps":true```

maybe we can use some renderer which will allow us to take screenshot or to be able to point it directly to displaytable used for VC map

-- edit

i believe that place where we can get real picture is function "NvMediaVideoDecoderRender" inside libnvmedia.so based on https://github.com/NakhyunKim/carProject/blob/master/include/nvmedia.h

[OneB1t]

anyway google maps icon rendering up&running i think that this week we can have fully working next-turn navigation data in VC

[grajen3]

For the native VC and next turn - I will have to figure out mapping from android auto next turn events to VC/BAP events

I did one ride with my scanner and on that I captured just those values:

   Field: turnSide
      "LEFT"
      "RIGHT"
      "UNSPECIFIED"
   Field: turnNumber
      "0"
      "90"
   Field: turnAngle
      "0"
      "90"
   Field: event
      "DEPART"
      "DESTINATION"
      "ROUNDABOUT_ENTER_AND_EXIT"
      "SLIGHT_TURN"
      "TURN"
      "UNKNOWN"

looking at https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/org/dsi/ifc/androidauto2/Constants.java#L120-L140 there is plenty more events that are possible.

turnSide and turnAngle seems weird to me - it probably meant I had just 90 degree left or right turns? but then I also had SLIGHT_TURN and TURN events so there will be need to get some understanding of that information :D I think I will do more inspection on event by event basis to understand that - I have trace files saved from that trip, so I can just log all individual events or even just search them up manually. Alternatively it might be possible with various android API public documentation (?) tho ones I see now have different format so will see about that.

The more interesting part will be mapping "AA events" to VC/BAP events - https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/generated/navsd/serializer/ManeuverDescriptor_Status%24Maneuver_1.java#L13-L47 unfortunately optimzer compiled away the values and inlined them in their place of usage, so I will have to do trial and error checking various integer values and see what is rendered and/or run actual navigation and capture used "main_element" values while checking what is being displayed

From local checking those MAIN_ELEMENT ints might be just consecutive numbers - just start nav in my garage where i was getting "calculating" and "offroad" displayed I got 9 and 6 integers so mapping that to MAIN_ELEMENT_CALC_ROUTE and MAIN_ELEMENT_OFF_ROAD those are 9th and 6th const in that file respecitvely (in 0-based indexing). I will be using audio texts for debugging on my next ride (+ logging): (D = direction, M = main element, Z = z level, S = sidestreets in https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/navsd/functions/ManeuverDescriptor.java#L71-L74 )

[jilleb]

Great work! I'll check the HUD-assets from the MIB2, there are a lot of graphics for each of the corner-angles and types of exits and such. I'll report back asap.

edit, the assets from Audi navigation has all the different angles: navigation_map.zip

I'll look for representation with better names/angles. I see many weird turns I never knew, like the Michigan turn 😮

edit 2: Lamborghini has some more logical names, including slight lane changes and turns.

I'll try to map the graphics to the names in the constants.java, for a better understanding

[jilleb]

Scratch the post above.. this is more useful, from the google maps API:

From the looks of it, there's a navigationturnside_left or _right, which determines the direction, followed by the type of turn. For instance, turnside left + slight turn:

Then we have left: UNKNOWN DEPART NAME_CHANGE ON_RAMP ROUNDABOUT_EXIT ROUNDABOUT_ENTER_AND_EXIT DESTINATION

[OneB1t]

char * FUN_001dd76c(undefined4 param_1)

{ switch(param_1) { case 0: return "UNKNOWN"; case 1: return "DEPART"; case 2: return "NAME_CHANGE"; case 3: return "SLIGHT_TURN"; case 4: return "TURN"; case 5: return "SHARP_TURN"; case 6: return "U_TURN"; case 7: return "ON_RAMP"; case 8: return "OFF_RAMP"; case 9: return "FORK"; case 10: return "MERGE"; case 0xb: return "ROUNDABOUT_ENTER"; case 0xc: return "ROUNDABOUT_EXIT"; case 0xd: return "ROUNDABOUT_ENTER_AND_EXIT"; case 0xe: return "STRAIGHT"; case 0xf: break; case 0x10: return "FERRY_BOAT"; case 0x11: return "FERRY_TRAIN"; case 0x12: break; case 0x13: return "DESTINATION"; } return ""; }

options from GAL

and all converted icons from AA https://github.com/OneB1t/VcMOSTRenderMqb/tree/main/icons

[OneB1t]

@grajen3 can you please upload this trace log somewhere so i see all combinations also with distance measurements?

Thanks

[grajen3]

@OneB1t extracted just nav events from my trip with roads/street names redacted a bit :) - https://gist.github.com/grajen3/40a06bdc5d2e370a28c1aef72346aed9 - note that I did end navigation 2 times:

• first time would be line 2849 and after restarting it to same place it was "straight line" to destination again
• second time would be line 6193 (note that I didn't reach my previous destination as I only set town/village as distanation and that point wasn't actually on my route - 3rd navigation was one that I did actually reach my destination, tho at this point I don't remember if I manually cancelled navigation those 50meters before it or not)

I do have 2 other trips that I forgot about, so I did also full scan on every trace I did have to collect distinct combination of next turn info (minus road) and:

   Field: DistinctAtrrbitesWithoutRoad
      "event: DEPART turnSide: UNSPECIFIED turnAngle: 0 turnNumber: 0"
      "event: DESTINATION turnSide: RIGHT turnAngle: 0 turnNumber: 0"
      "event: ROUNDABOUT_ENTER_AND_EXIT turnSide: RIGHT turnAngle: 180 turnNumber: 180"
      "event: ROUNDABOUT_ENTER_AND_EXIT turnSide: RIGHT turnAngle: 270 turnNumber: 270"
      "event: ROUNDABOUT_ENTER_AND_EXIT turnSide: RIGHT turnAngle: 90 turnNumber: 90"
      "event: ROUNDABOUT_EXIT turnSide: RIGHT turnAngle: 0 turnNumber: 0"
      "event: SLIGHT_TURN turnSide: RIGHT turnAngle: 0 turnNumber: 0"
      "event: TURN turnSide: LEFT turnAngle: 0 turnNumber: 0"
      "event: TURN turnSide: RIGHT turnAngle: 0 turnNumber: 0"
      "event: UNKNOWN turnSide: UNSPECIFIED turnAngle: 0 turnNumber: 0"

some notes here:

• turnAngle and turnNumber might be ROUNDABOUT_* only thing. I'm in right-hand traffic country - so ROUNDABOUT_* turnSide=RIGHT seems to be indication of that (plus of course that would have to change icon in a way that regular TURN (and its variants) wouldn't). I only been on roundabouts with 4 exits each 90 degrees apart - so dunno if there are more specific angles for less usual (at least for me) roundabout layouts.
• ROUNDABOUT_EXIT might be when I either don't listen to navigation route or on "bigger" ones 🤷
• TURN and SLIGHT_TURN doesn't seem to have angles - just turnside
• DEPART seems to be event that happens when google calculates your route and road is name of road/location you currently are in
• In my 3 trips that I had - destination was always on the right - will see if it will show LEFT in the future
• I will be having longer trip tomorrow with entering/exiting highways, ramps etc - so hopefully that will allow collecting much more things :)

This is for newer APIs on what navigation apps should report to android (which in turns feeds android auto?) - some of the events maybe we could get description from here as well: https://developer.android.com/reference/androidx/car/app/navigation/model/Maneuver#constants_1

[grajen3]

Some minor updates for using native VC route guidance UI elements:

1. Route guidance status need to be reported as active - you do this by setting https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/navsd/functions/RGStatus.java#L54 to 1
2. Setting those https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/navsd/functions/ManeuverDescriptor.java#L127-L130 feed into "next turn" info (at least in my right gauge cluster)

I just hardcoded data I recorded when sitting in my garage to force render without any route guidance active to achieve same result I got when running native route guidance and that was succesful - tho it was just "offroad" stuff I shared before, but at least I managed to show it myself - this replicated what I shown on photo in this comment https://github.com/jilleb/mib2-toolbox/issues/159#issuecomment-1870955547 (minus the distance part, I didn't mess with that yet)

(In RGStatus.java)

    private void setRouteGuidanceStatus(RG_Status_Status rG_Status_Status) {
        rG_Status_Status.rg_Status = 1;
    }

(In ManeuverDescriptor.java)

    private ManeuverDescriptor_Status computeManeuverDescriptorStatus() {
        ManeuverDescriptor_Status maneuverDescriptor_Status = this.dequeueBAPEntity();
        if (this.getNavigationService().getRouteGuidanceState() != 0) {
            this.fillManeuverDescriptor(maneuverDescriptor_Status);
        } else {
            maneuverDescriptor_Status.maneuver_1.direction = 96;
            maneuverDescriptor_Status.maneuver_1.mainElement = 6;
            maneuverDescriptor_Status.maneuver_1.sidestreets.setContent("test"); // this was not shown - it did say "Offroad" in VC which was what `mainElement = 6` is for I think (`MAIN_ELEMENT_OFF_ROAD`) - I log/display it in audio tab so will see if this is used in TURN etc events
            maneuverDescriptor_Status.maneuver_1.zLevelGuidance = 0;
        }
        return maneuverDescriptor_Status;
    }

I still am before recording native route guidance data for a longer trip (well in fact any actual trip, so far I recorded only stuff without moving at all) to get data to come up with some mapping with both Android Auto and native route guidance at the same time to hopefully correlate them (and hope that calculated route for both would be the same one :D). I imagine I could do try and error without that, but I'm also bit weary trying to feed it "random" stuff that it might not expect/know how to handle - so this is just slight risk reduction heh. I also will use Audio tab to display values of maneuverDescriptor_Status.maneuver_1 and DistanceToNextManeuver stuff live (ideally I would video record my VC with debug info + what is shown in right gauge, but don't have setup for that lol)

Some changes need to be made in code I linked and my line links above are just to make it more understandable what fields exactly need to be set - process method is entry point from where updating VC data/BAP starts (and calling it is how I force texts to update in VC) and eventually native implementation reach those lines that I linked to - later there likely will be 2 seperate code paths in process methods (pseudocode):

if (androidRouteGuidanceActive) {
  // always priortize android route guidance (?) possibly could be other way around as native probably be more "accurate" at least in terms of mapping native events to VC stuff than our custom android handling (not the accuracy of navigation itself, just accuracy of mapping events) - but this would only matter if both navigations are active
  handleAndroidData();
} else if (nativeRouteGuidanceActive) {
  handleDefaultImplementation();
}

[grajen3]

I've made some progress on figuring out some internals of

• ManeuverDescriptor
• DistanceToNextManeuver
• TurnToInfo

ManeuverDescriptor:

• direction is int between 0 and 255 and it maps more or less to 0-360 degree (counter clockwise), so:
  • direction = 32 -> it's 45 degress (left)
  • direction = 64 -> 90 degrees and so on
• mainElement mapping is what I suspected with 0-based values from order in https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/generated/navsd/serializer/ManeuverDescriptor_Status%24Maneuver_1.java#L13-L47 so
  • mainElement = 0 -> no symbol
  • mainElement = 1 -> no info
  • mainELement = 2 -> direction to destination and so on
• some mainElements can also have 0 direction (otherwise icon disapear), some elements can have direction but only select few values are "supported" and icon is displayed - i.e. direction=16 does show on roundabout icon, but on regular turn it doesn't show icon at all
• sideStreets is not string that is displayed - this is some weird encoded stuff to show other streets to give context to which street you should turn when there is multiple intersections / multiple actual streets you could turn right/left to - I didn't attempt to decipher it but logged values are things like that: ÿ\0AÁ etc - It probably should be presented as string and it would make more sense this way, but 🤷 it's string in decompiled java code - we also are not getting any info from android auto about that - so this will likely be unuse

DistanceToNextManeuver:

• unit 0 = meters, unit 1 = km - imperial units N/A for me (probably could switch units, to figure it out, but for now I want to get something working for me :D)
• distance is distance in selected unit * 10:
  • so 1000m can be presented either as distance=10000 (1000 10), unit = 0 (meters) OR as 1km with distance=10 (1 10) and unit = 1 (km) TurnToInfo DOES show in right gauge cluster above maneuver icon- but it depends on mainElement in ManeuverDescriptor - not all elements show it

I will be doing some better way of figuring out all valid combination (or maybe just combinations of things needed to replicate all needed icons as indicated by icons @OneB1t pushed to https://github.com/OneB1t/VcMOSTRenderMqb/tree/main/icons

but just some examples: (first row in center D<direction> M<mainElement> <GUESSED_MAIN_ELEMENT_DESCRIPTION> (second row in center D<distance> U<unit> ...rest is for bargraph testing which wasn't really succesful for me yet :D)

(above one is example where TurnToInfo is not displayed)

[OneB1t]

that is very nice progress i will try to replicate that when i have some free time in my car

[grajen3]

things are slowly moving forward - managed to get this (simulating possible android auto events): https://www.youtube.com/watch?v=dmpFfU2WC5U

there are couple of problems that I didn't get ironed out before my car battery started to die - need to take a ride to charge it a bit heh, possibly fixed them already in my mess :)

there is no good icon/element for merges or ramps (or ferries or trains, but I didn't even bother with those as I never use those anyway - probably will just use destination icon for those)

Part I did figure out since last update - bargraphs (on video linked above you can see those in action when it says 500m - the bar is "full" and it goes down when distance goes toward 0). In https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/de/vw/mib/bap/mqbab2/navsd/functions/DistanceToNextManeuver.java :

• to enable bargraph mode - distanceToNextManeuver_Status.bargraphInfo.bargraphOnOff need to be 1 (0 would be off) - in my video it's 0 when distance > 500m - you can see icon does change slightly on 500m mark)

• to control how filled the bargraph is you use distanceToNextManeuver_Status.bargraphInfo.bargraph (duh) - 100 is fully filled (that's what I set at 500m) and 0 is "empty" - in general this snippet is what I use to do both distance text and bargraph:

          if (AndroidDistanceToNextManeuver < 1000) {
              // display meters when below 1km
              distanceToNextManeuver_Status.distanceToNextManeuver.distance = AndroidDistanceToNextManeuver * 10;
              distanceToNextManeuver_Status.distanceToNextManeuver.unit = 0;
          } else {
              // display kilometers when 1km and above
              distanceToNextManeuver_Status.distanceToNextManeuver.distance = AndroidDistanceToNextManeuver / 100;
              distanceToNextManeuver_Status.distanceToNextManeuver.unit = 1;
          }
  
          if (AndroidDistanceToNextManeuver <= 500) {
              // start showing bargraph at 500m meters mark
              distanceToNextManeuver_Status.bargraphInfo.bargraphOnOff = 1;
              // just scale it linearly with distance
              distanceToNextManeuver_Status.bargraphInfo.bargraph = AndroidDistanceToNextManeuver / 5;
          } else {
              distanceToNextManeuver_Status.bargraphInfo.bargraphOnOff = 0;
              distanceToNextManeuver_Status.bargraphInfo.bargraph = 0;
          }
  
          distanceToNextManeuver_Status.validityInformation.distanceToNextManeuverValid = true;

I also think I should just show "follow along" icon when next maneuver is X kilometers and/or Y minutes away and only swap to actual maneuver when I'm reasonably close enough to start actually thinking about it - on one of my trips android told me that next nav event is to turn right ... in 95 kilometers - which was correct, but I would feel weird seeing turn right icon in my gauge cluster for that long

I will still need to do more changes to my scanner/parser because on long enough trips it takes too long to parse things (but I do full re-parse of last log file now, so I will see if I can avoid that and just try to read "new bytes" since last attempt, should be possible but I'm java newbie and googled solutions often times are not applicable because of subset of java ~std lib that is available to use in lsd)

-- edit slight annoyance I have is that setting RGStatus to 1 automatically switch VC to map view - maybe if I had custom dashboard renderer it would be less of a problem (because I would be on that dashboard anyway :D)

[jilleb]

Awesome work man! Let me know if I can assist or test anything on the Leon virtual cockpit.

Edit: I forgot about exitview.. not sure if it's in use on the AID, or if it's a HUD thing.

[OneB1t]

maybe if I had custom dashboard renderer it would be less of a problem (because I would be on that dashboard anyway :D)

I think it can be much better as you can also show calculated seconds until next turn based on current speed or you can also draw parts in red if you are close etc.. and it is possible to run it and see result at home in IDEA without going to car which is very helpfull as it should get -10 this week here :-D also i like to see my current/avg consumption in left cluster and range in right one

[jilleb]

I had a look at /de/vw/mib/bap/mqbab2/navsd/functions/ It has the following additional navi functions, and I believe they are there on some brands/virtual cockpits, not sure if we can use them as well:

Exitview.java Showing the off-ramp with ramp-number/name. Not sure if this info is available on Android Auto/Carplay though..

LaneGuidance.java Shows which lane to drive on.

Altitude.java VC can show the altitude, maybe we can feed it too? Not the most useful data, but still.. it's missing on CP/AA

CompassInfo.java The compass showing the current heading of the car. Shown when no route is being followed.

Functionlist.java has some other functions, some disabled on VW but I've seen them available on other brands.

[grajen3]

Let me know if I can assist or test anything on the Leon virtual cockpit.

I'll try to get enough things working to get to a place I'm happy with and clean up my code, push that and share .jar from it (for now I have single jar together with original NavIgnore patches and probably will keep it that way before I start thinking on what's the best way to have those separate).

I think it can be much better as you can also show calculated seconds until next turn based on current speed

You can do that with bargraph stuff to some extent - gal also already gives you timeSeconds together with distanceMeters to next turn event (tho I will use distance and not time for bargraph anyway as it would be weird if bar starts growing in case of traffic jam hah)

it is possible to run it and see result at home in IDEA without going to car which is very helpfull as it should get -10 this week here :-D

Yeah - that's the most annoying part of this :D I have car in garage so it's not that cold, but it's still not very nice to try to make changes in your car and I did get "battery low" and everything turned off couple of times already :D I hate connecting charger so I then pause my tinkering until I have reason to drive somewhere to recharge battery a bit before I do another session :)

also i like to see my current/avg consumption in left cluster and range in right one

I don't need to see those to know that my consumption is too high and range is too low :D Just kidding - for me it's ACC distance to next car on the left (most of the time I travel is on highways and seeing if radar actually sees car in front of me gives me some peace of mind), driving data in middle (oil temp usually, but will scroll to phone sometimes or even pretty zoomed in map to get some sense of how sharp next corner will be when doing more ... spirited driving :) ) and right is compass/nav just because that's what preset for left/right gauge is and can't customise those to my liking (unless you somehow can control them programatically individually that is just not possible with controls we are given) - so might as well actually use that for android auto nav if I'm in this view anyway.

But generally playing with some custom dashboard might be next project after I'm finished with what I'm doing now. But really, unless we could mirror android auto on there, I don't think I will get much practical value of using custom dashboard (well, maybe if someone build one that I like then yeah, but probably would not be willing to start another "big" project anytime soon if I would like to build one that I'd like to use - this stuff already became a bit of time sink haha).

PS. I did try https://github.com/grajen3/mib2-lsd-patching/blob/aa-in-vc/lsd_java/CDisplaySPITest.java and it did capture nav map from VC (regardless of view I was on) and regular head unit UI, but when Android Auto was shown I got blank image (kind of what I expected but decided to try anyway) - it did complain than image was not "expected" 800x480 :D (both main head unit display and VC is 1280x640)

Exitview.java Showing the off-ramp with ramp-number/name. Not sure if this info is available on Android Auto/Carplay though..

This is the only recorded actual "ramp" event I have from gal (on my pc, might have more on sd card as I didn't pull newer traces lately) before entering highway from regular road ("A4" is highway designation):

road: A4 turnSide: RIGHT event: ON_RAMP turnAngle: 0 turnNumber: 0

LaneGuidance.java Shows which lane to drive on.

We don't get those with our version of gal - those are available in newer ones potentially ( https://developer.android.com/reference/androidx/car/app/navigation/model/LaneDirection is for Android Auto navigation apps builder) - maybe there would be tricks to extract that information out of android device and pass it back to car, but I dunno how (possibly could be figured out by decompiling android auto .apk, but that's whole new project to get access to those). https://developers.google.com/maps/documentation/navigation/android-sdk/tbt-feed?hl=en is I think about getting access to those but it's behind "Contact sales" so not really a public API

Altitude.java VC can show the altitude, maybe we can feed it too? Not the most useful data, but still.. it's missing on CP/AA

This is shown on mine in "navigation" preset in left gauge (and compass/nav info on the right)

CompassInfo.java The compass showing the current heading of the car. Shown when no route is being followed.

Yup shows in couple of presets when RGStatus is 0, but it has only angle and direction_Symbolic (which probably maps to N, S, W, E, NW, NE, SW and SE)

[OneB1t]

Android Auto was shown I got blank image (kind of what I expected but decided to try anyway) - it did complain than image was not "expected" 800x480 :D (both main head unit display and VC is 1280x640)

Actually the map stream is 800×480 only for this generation of cockpit

Maybe there is some car which have never version of gal binary and we can try to copy it to our unit

[jilleb]

what preset for left/right gauge is and can't customise those to my liking (unless you somehow can control them programatically individually that is just not possible with controls we are given)

The Seat and Skoda VC have this instrument setup screen for this, so it can be finetuned. really weird that VW didn't put it in their clusters.

[OneB1t]

what preset for left/right gauge is and can't customise those to my liking (unless you somehow can control them programatically individually that is just not possible with controls we are given)

The Seat and Skoda VC have this instrument setup screen for this, so it can be finetuned. really weird that VW didn't put it in their clusters.

It is available for gen2 clusters (10") there you can set it up in infotainment

[andrewleech]

I'm not sure what gen/size my cluster is to be honest (2019 VW golf Alltrack), if you can give some more detail on where the settings are I'll take a look.

I'm loving the update feed too from you both, this is already much further along than I expected possible.

I'm certainly happy to help with organising / structuring jar files when there's something to share if you'd like.

They being said I'm busy with some house projects getting ready for a soon to be larger family so haven't had any time to really get involved again lately!

[grajen3]

I'm not sure what gen/size my cluster is to be honest (2019 VW golf Alltrack), if you can give some more detail on where the settings are I'll take a look.

https://dx.dragan.ba/aid/ this shows how gen1/2 looks like - first photo is of gen 2 + customization screen in head unit. Gen 1 doesn't have that customization and only can toggle between presets in "view" tab displayed in VC itself (at least with controls that are available - maybe we can hack something in :D).

I think gen 2 also has fuel/coolant temp gauges actually outside of VC itself

[jilleb]

That's indeed what's in the Leon as well. There are some third party hardware solutions that can display additional information on the media screen of the AID, but I think we can now so the same without the need for any hardware mods 😁

[andrewleech]

Ah yes, it seems I've got the more basic one :-(

[OneB1t]

i think it is like this:

VW: golf: only gen 1 arteon: both gen 1 and gen 2 passat: both gen 1 and gen 2 tiguan 2: both gen 1 and gen 2 vw atlas: both gen 1 and gen 2

skoda/seat: only gen 2

[grajen3]

Hey folks, I pushed somewhat cleanup stuff in https://github.com/grajen3/mib2-lsd-patching/commits/aa-in-vc/ - Had short ride yesterday with it and was quite pleased (other than "straight" on roundabout shown as "uturn" in VC :D, but I think I fixed that part today, but didn't verify yet) - some stuff that was changed since my last push:

• when scanner reach end of last trace file, it will not try to rescan that file from the start and just try to read new bytes - with 0.5s sleep after it reached EOF, so should be fast enough - it also does scan if there is new trace file/shard when it reached EOF at the same time to know wether it should be done with current trace file)
• most importantly handling of nav traces - I tried to get rid of "magic numbers" and introduce my own constants for readability for others - https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/patched/de/esolutions/fw/comm/dsi/androidauto2/impl/DSIAndroidAuto2ReplyService.java#L843-L963 I hope this "mapping" is somewhat readable for folks other than me now
• I tried to handle left-hand drive situation to best of my guessing game (this really only seems to matter for roundabouts) - but I can't test this in practice, so your mileage may vary with my code
• I didn't handle imperial units (miles etc) - just metric (kilometers) + just from observing google maps on headunit I added some rounding of distance that is displayed on VC to hopefully match what projected google maps on head unit show. I did leave comments / thoughts / guesses on imperial units in https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/patched/de/vw/mib/bap/mqbab2/navsd/functions/DistanceToNextManeuver.java#L153-L199 (both how to probably detect which units to use and "constants" for units themselves) but because I'm not used to those units I didn't even attempt to implement handling of it

https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/patched/NavActiveIgnore.jar is updated .jar if you don't have setup completed

I have a note about Audio stuff: https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/patched/de/vw/mib/bap/mqbab2/audiosd/functions/CurrentStationInfo.java#L754-L762 this has a timer I guess to prevent trying to push data to frequently - sometimes when track change (either just when it naturally ends or you force next/prev track - it wouldn't show update in VC when you are actively on Audio tab/view - switching views back and forth seems to refresh data - I didn't really debug it, tho in my dirty branch where I do explorations I got rid of the timer and did force updates to just happen and it seemed to get stuck less frequently, but it still would get stuck occassionally. https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/patched/de/vw/mib/bap/mqbab2/audiosd/functions/CurrentStationInfo.java#L680 is what I assume sends information and it does return boolean, so maybe that would indicate wether update was handled/accepted and maybe some tinkering there would allow actually handling situations where it currently does get stuck. (I did have this issue from the start and was sometimes frustrating when trying to use audio tab/view when displaying my debug info, but because there was workaround with changing view briefly, I didn't dig into it for now, but now that nav is somewhat working - this might be something for me to try to dig into)

I did look into CoverArt as well, but so far wasn't succesful, however few things for that part:

• https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/lsd_java/de/esolutions/fw/comm/dsi/kombipictureserver/impl/DSIKombiPictureServerProxy.java - those responseX things are logged when using radio/bluetooth/sdcard
• they are triggered from here https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/lsd_java/de/vw/mib/asl/internal/kombipictureserver/usecaces/CoverArt.java
• CoverArt uses bapAudioSdService which is implemented by https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/lsd_java/de/vw/mib/asl/internal/kombipictureserver/common/api/bap/BapAudioSdAdapter.java to know "current source" and "current position of track"
• BapAudioSdAdapter get information from:
  • https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/lsd_java/de/vw/mib/bap/mqbab2/audiosd/functions/CurrentStationHandle.java for getCurrentStationHandleCurrentStationHandle (this is like position in playlist - changing it result in animation in vc where album/cover art moves to make place to new one - this is also passed to responseCoverArt as well
  • https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/lsd_java/de/vw/mib/bap/mqbab2/audiosd/functions/ActiveSource.java for getActiveSourceSourceType
  • those are used here https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/lsd_java/de/vw/mib/asl/internal/kombipictureserver/usecaces/CoverArt.java#L141
• CoverArt also uses https://github.com/grajen3/mib2-lsd-patching/blob/0c0981b3568309f903dcbf973b86fdc49302c02e/lsd_java/de/vw/mib/asl/internal/kombipictureserver/common/api/media/MediaCoverArtAdapter.java to get actual ResourceLocator (which is object with url string field that would point to /tmp/gal_albumArt_X.png

In my tests I was able to make CurrentStationHandle update playlist position and trigger CoverArt to actually do stuff like this: DSIKombiPictureServerProxy.responseCoverArt(2, 0, 19, 1, ResourceLocator(Type isURL; Value=/tmp/gal_albumArt_2.png)) (19 seem to be "sourceType" for android auto audio - I've been loging bunch of stuff, and I don't actually hardcode that - I let native provide that - I only override MediaCoverArtAdapter ResourceLocator getter when sourceType is 19. I did make CurrentStationHandle to update "position in playlist" when we get new cover arts from android auto (otherwise it would be -1 which seems to mark stuff as "invalid"). However I still am not actually getting cover arts displayed - tho I do get animation of "default cover art" image when changing tracks/albums