Phone call video focus issue discussion

[lmagder]

This isn't a bug per-se but github doesn't have forums. I figured we should channel all the discussion about fixing the phone call video focus stuff in one place instead of random commit comments etc., even if we don't have a PR yet.

[lmagder]

As a status report I tried hacking up a simple test on my Linux PC to prototype the idea from @mishaaq about DBus filters since it's way easier to test and debug than the car and it seems like at this point it's mostly DBus we are trying to figure out: https://github.com/lmagder/dbus-intercept-test/tree/master/server

I got the filter working using the "eavesdrop" mode however it doesn't seem possible to edit or consume the messages, only view them.

Next I tried registering a wrapper implementation for the DBus path/interface and then delegating to the original object by referencing it by the raw unique ID (since the wrapper took over the path). This works great actually (you can try it in the repo) but the issue is that it requires the original object to specify the DBUS_NAME_FLAG_ALLOW_REPLACEMENT when it's registered. Maybe the car does this (you can always dream :smiley:) but probably it does not, but maybe we can hack it on somehow. I dunno.

EDIT: Sorry forgot to include my actual question lol. Any ideas how to get the car to pass DBUS_NAME_FLAG_ALLOW_REPLACEMENT when creating the DBus object or for ways around needing this flag?

[Trevelopment]

I don't know about that DBus stuff but I wanted to share some things I tried. Since I'm a C++ n00b I am just kind of trying simple things that may be a little bit hacky like pulling the if(isPressed){callbacks->takeVideoFocus();} in mazda/outputs.cpp outside of the switch statement (meaning every button press requests video focus.)
Of course this does work in a really hacky kind of way but here is the interesting thing I didn't expect, when I hit the home button well duh, it jumps back into the AA screen because it is releasing=>requesting focus when you press it BUT when I would get the BLACK SCREEN I hit the home button and BOOM, AA screen comes up every time right away. So there might be something there in terms of that black screen issue because I put in my nav SD and another USB with lots of music just to force a memory error and it still worked to bring up the AA screen after getting the black screen in all my tests so far. I think that it is really gstreamer-0.10 that is the problem in those situations but Do you know if it would be at all possible to upgrade to gstreamer-1.0? I don't think it would help us with this issue in particular but I am just wondering if is is viable to do that and if it would help at all? I was also thinking maybe changing some of the JS code might make it work better so I have been hacking around in that area too and I will let you know if I find anything promising.

[Trevelopment]

Sorry didnt mean to close it 🤕

[lmagder]

That's interesting losing and getting back video focus does re-initialize gstreamer since when the focus is lot is shuts down the library. However if it already is running it just sends the message to the car again to start sending the video stream, so I'm not sure how that is related to the sdcard. I don't have the nav sdcard so I can't test, I'm just going off of what you guys say. Maybe losing and getting focus again resets gstreamer and the second time it works? I dunno.

It definitely could be something wrong (memory leaks) with the version of gstreamer on the car since it is old and they have a bunch of custom plugins installed but I'm not sure it's possible to upgrade. The 0.10 version on there is part of the CMU and even if we statically linked or distributed a private copy of 1.0, that wouldn't be compatible with the custom non-open-source plugins they have on the car to use the hardware video decoder and output to the video overlay surface, specifically the vpudec and mfw_isink elements which are made by some company called NXP and are not publicly available AFAIK

Maybe the CMU is fast enough to do entirely CPU h264 decode? I've never tried. In that case probably using a static-linked x264 lib directly is the way to go instead of the whole gstreamer since our task is very simple and doesn't require dealing with any containers or av/sync etc, format conversions, etc. The only reason why the code uses gstreamer now is that it's the only way of accessing the proprietary video hardware. It used to also use gstreamer for audio also but was using the also output node and not much else so I converted the code to use libasound directly since playing a simple uncompressed PCM stream really doesn't need the gstreamer infrastructure.

[Trevelopment]

That's all very cool stuff. have you tried the Mazda VideoPlayer app? https://github.com/Trevelopment/Mazda-Videoplayer I did some major upgrades on it and learned some things about gstreamer and memory in the CMU. I found it handles h264 video format like a champ (the best format being 360p MP4 H264 AAC) and it uses gplay which is a little probably custom app built on top of gstreamer and it works pretty well. I don't know maybe you can use that in some way it has its own key mapped controls video player uses websockets to use them but you could probably communicate directly with it. I dunno is that useful at all? I kinda feel like I veered off topic a little.

And all you need to know about the Nav SD cards is this: they are trying to process way too much information at once and they eat up memory like pac man eats dots. I made a little Memory Logger and using that I saw that the Nav system will just keeps chowing down on your memory and as time progresses you will eventually reach an inevidable memory crash. I guess it's not that dramatic and I did test it while running a bunch of other things running at once like speedometer and AA and videoplayer but that's besides the point that the CMU should have been built with double the memory it has.

Here's a funny side note I went out to my car to test if the that still worked with the black screen issue and it took 2 presses this time but the CEO of my company came up to my car window and I just unloaded a cloud of weed smoke in his face when I opened the window.

[viktorgino]

@lmagder gstreamer 1.0 i.MX6 plugins are open source, you can find them here. From what I can tell you can use these plugins on your head units. You should be able to replace vpudec with imxvpudec and mfw_isink with (I think) imxg2dvideosink

[lmagder]

@viktorgino Nice, I guess it could be possible then. Although I noticed that depends on https://github.com/Freescale/libimxvpuapi, which is OSS but not sure about the dependencies of that. It seems to need other freescale libs, but maybe there are also out there somewhere else. It could be OSS all the way down to the kernel-side due to GPL viral-ness.

But either way building a version of gstreamer 1.0 we could package as private to the app or adapting the code to use the libimxvpuapi library directly are a lot of work. Especially since we don't know if it would fix stuff like https://github.com/gartnera/headunit/issues/72 without trying it or if the kernel interfaces have changed in an incompatible way since the version on the car.

I don't think it would help us with the video focus issue though unless we could somehow bypass the entire SetRequiredSurfaces() and window manager completely.

[Trevelopment]

@lmagder I got something working pretty good on the JS side. Its pretty hacky but I got it working for at least the beginning of the calls very consistently. There are just 2 stipulations about it: 1. it is a function that I included in _androidauto.js (because it requests video focus with AAcallCommandServer function) so you have to open Android Auto from the applications menu for it to work. This could be fixed by loading the 'global' AA functions at startup and also calling this function at startup (it uses setInterval so it will run in the background until its conditions are all true meaning [ the current app is phone && ActiveCall is current context && AA is connected && doesn't have video focus] and we can tweak what it is looking for as needed and I check the app contexts first so it doesn't make an http request every second) 2. You do get the MZD phone screen pop up for like 1 second every time like we talked about before. I could lower the interval time to like half a second but I think that it needs that extra second to finish changing contexts or else it 'blows out' the surface change. This is a pretty good temporary solution like my Bluetooth call patch was and maybe, similarly it might give you an idea of what needs to be done on the C++ side. So yea we should still try and intercept the DBus message if we can pull that off but this way does work. I'm going to test it a little more, polish it up and Ill push it to a new branch in the repo so you can check it out. One funny thing is that if AA is connected and you use regular MZD phone app you end up on the AA phone screen but I don't think that is a very big deal.

[Trevelopment]

89