search

rhysmorgan134 / node-CarPlay

MIT License
138 stars 24 forks source link

Android Auto flickering screen #11

Closed Tigo2000 closed 8 months ago

Tigo2000 commented 1 year ago

When plugging in an Android phone into the carlinkit adapter with this image (which uses node-carplay), the android auto projection starts up but the screen flickers, this seems to be on the same cycle as the "requesting key frame" messages. The CarPlay projection does not seem to be flickering.

Here is a video where you can see it happening. https://youtu.be/hBsnDfqHGdA (Flickering starts at 0:10, you can see the logs at 0:40)

Do you happen to know what causes this? I have also tested this on version 1.0.5 of react-carplay and the screen does not seem to flicker on that release.

I've added a console.log statement in the componentDidMount() function to test whether the component but this does not seem to be the case. This function is only fired on opening the carplay page, not when plugging in a phone or while the screen is flickering.

If you need any more info please let me know.

Running this on a Raspberry Pi 4B.

rhysmorgan134 commented 1 year ago

Thats interesting, did it work with android auto out of the box? I haven't tried it with android auto, so currently I would have to try and dig an old phone out and give it ago.

If it worked out of the box, I would suggest giving react-carplay a test and see if you get the same symptoms there

Tigo2000 commented 1 year ago

Yeah, Android Auto worked out of the box! The dongles support Android Auto so I figured I'd just try it out. Only testing wired now, but wireless should work in theory as well.

The react-carplay image works well with Android already (tested), it's just this node-carplay image that seems to be the problem. I spoke to LRYMND and he mentioned that the react-carplay image itself was using a newer version of react-js-carplay than the version that was available in the react-js-carplay repo at the time of creating this issue. Since npm only installs the latest version of react-js-carplay that is available in the public repo (and not the one in react-carplay), there's a version difference which could explain the flickering issue. You seemed to have updated the repo with this commit which has fixed most of the differences though :)

LRYMND is going to implement the rest of the changes from the newest react-js-carplay and then we'll see if android works without flickering, would be nice to have it working!

Do let me know if you need me to test anything or provide logs.

Thanks for your work!

rhysmorgan134 commented 1 year ago

That’s good to know, Thankyou! I suspected android needed atleast a different phone type in the protocol but apparently not!

it sounds like your issue resides in LRYMNDs project, it is designed to be used with react-js-CarPlay library as the component, so uses of modified version may well cause conflicts upon updates!

Tigo2000 commented 1 year ago

Hello Rhys, Sadly I may have celebrated too early. Just tested react-carplay and the node-carplay based Volvo-RTVI image with the newer 1521-dongle, and sadly on these different images (with a wired connection on the 1521-dongle) the Android phone is not recognized at all. On my old dongle (1314:1520), Android Auto does seem to work on the react-carplay image as mentioned earlier in this issue.

I have tested all possible combinations and logged the console output to a text file.
log messages.zip

As you mentioned Android Auto might need a different phone type in the protocol after all, but you might be able to figure it out using the log files.

It would be great if we could figure it out as this would be a great addition to both node- and react-carplay projects.

Thank you in advance!

kerta1n commented 12 months ago

+1 for this, when trying to run on the :3000 webserver, AA isn't recognized at all (sorry for commenting so late)

steelbrain commented 12 months ago

Digging into this today, with some Android devices I have lying around on the next branch, let's see if it's still relevant

steelbrain commented 12 months ago

Did a bunch of experiments with a Asus RogPhone and a Samsung Z Flip 4, here's the results

You can find some references to this bug on Reddit, ie https://old.reddit.com/r/AndroidAuto/comments/hq5b2o/android_10_screen_flickers_and_glitches_out_once/

gozmanyoni commented 11 months ago

@steelbrain there are some AA specific settings in the APK related to resolution and aspect ratios - I can add these to a branch based on next - dont have an android phone to test myself sadly

steelbrain commented 11 months ago

@gozmanyoni I have a bunch lying around, happy to test any hypothesis or code if you can give me some pointers

gozmanyoni commented 11 months ago

@steelbrain its a part of the SendBoxSettings message (we send the mediaDelay value there).

The apk also sends theses two props as part of the json body:

jSONObject.put("androidAutoSizeW", p.k.f1962b); jSONObject.put("androidAutoSizeH", p.k.f1963c);

if you extend the message and pass the resolution - It may solve it

gozmanyoni commented 11 months ago

At this point the resolution bug is fixed, and we have determined that flickering is in fact phone specific and not dongle related - should this issue be closed?

Tigo2000 commented 11 months ago

Thanks to everyone who worked on this for the last few weeks!

@gozmanyoni Good to hear that this doesn't seem related to the dongle, any hypothesis why/how it's phone specific? Android version / OS, chipset perhaps? I have tested with multiple OnePlus phones, mostly Android 11/12/13. (OxygenOS)

Can I test it myself by building https://github.com/rhysmorgan134/node-CarPlay/tree/next and using the most recent react-carplay version or are there any specifics to know? Got a few other phones lying around as well so happy to test this further!

Thanks!

rhysmorgan134 commented 11 months ago

You are best off testing using the bundled example in this branch

https://github.com/rhysmorgan134/node-CarPlay/tree/next/examples/carplay-web-app

the readme covers the install. I am working on a new react-carplay that uses this library

gozmanyoni commented 11 months ago

@Tigo2000 according to the reddit issue posted above it seems to be OS/Config related - one of the suggestions there is

go settings>developer options> Disable HW overlays

steelbrain commented 11 months ago

Just adding on to the above comments, if you'd like to run the bundled example, you can grab a built-and-ready package from any of the recent CI runs, ie (https://github.com/rhysmorgan134/node-CarPlay/actions/runs/6279417324) by downloading the artifact "carplay-web-app.tar.gz"

Tigo2000 commented 11 months ago

Thanks all for the suggestions and the help.

Spent some time testing Android Auto using the bundled example Rhys mentioned with a few different Android phones on a fresh install of Raspberry Pi OS Bullseye (64-bits), below are my findings:

The CCPA/CCPM dongles both work with iPhones. I have another CCPA dongle which is currently buried deep inside my car so testing with that dongle is difficult.

Is there some kind of timer/job/anything that currently runs at 5 seconds, which used to run every second which could perhaps explain the flashing in the youtube video?

Toggling "disable hw overlays" in the dev options didn't result in any noticeable difference.

In the video linked you can also see that the touch inputs do not line up with the carplay/AA screen, is this due to me entering the inputs through VNC rather than directly through mouse/kb? I did connect a screen to the pi and the inputs through VNC did line up with what was shown on the 'real' display.

Thanks!

PS. When unplugging an android/iPhone, the projection is still shown instead of going back to the spinning wheel. Is this expected behavior of node-carplay?

rhysmorgan134 commented 11 months ago

Thanks @Tigo2000. @gozmanyoni these times (1 second on react CarPlay, 5 seconds on new version) tie in exactly with the request keyframe timing that have changed.

I'll try and dig an old android out and give it a test, perhaps android automatically sends out keyframes.

@Tigo2000 if it's not too much of a problem, would you be able to list firmware versions for each of the dongles?

Tigo2000 commented 11 months ago

CPC200-CCPA: 2022.11.19.1218 CPC200-CCPM: 2022.04.25.1323CAY

react-carplay didn't show the version number for the third dongle which might be faulty.

gozmanyoni commented 11 months ago

Thanks for the info @Tigo2000

5000ms is indeed the interval we use to request a key-frame. but android might not need this.

To test this out I created this branch - https://github.com/rhysmorgan134/node-CarPlay/tree/feature/configurable-frame-interval

in this branch, if you can change the example and set the following config in App.tsx:

const config: Partial<DongleConfig> = { width, height, fps: 60, mediaDelay: 0, frameInterval: null }

Also make sure to run, in the main folder

npm install npm run build

and then in the example folder npm install npm start

This will disable the 5000ms keyFrame interval - it exists because on carplay, keyframe are only sent when requested, and without these jmuxer develops both a memory leak and a video delay.

Android may send these by default and not handle the frame request the same way as iOS.

gozmanyoni commented 11 months ago

@Tigo2000 @rhysmorgan134 - Also, on the same branch i pushed a fix for the loading/plugged logic in the example - did a small refactor and fixed the treatment for isPlugged.

steelbrain commented 11 months ago

@Tigo2000 It's really interesting that CPC200-CCPA is not working for you. I do my testing on this dongle and it works well. I do have the firmware from 2023 tho. Have you only tested with wired android auto or wireless as well? I've primarily only tested for wireless

Tigo2000 commented 11 months ago

@gozmanyoni Just tried out the new feature branch, i.e. added the frameIntervall: null to the (dongle)config in app.tsx and the flashing seems to be gone! 🎉 After unplugging the phone, the projection stops and it shows the spinning wheel again, so that fix is working as well.

After the initial test I rebooted and tried to listen to music again, but I don't have any audio output. In the dev console I get the errors/warnings below in the following order:

Failed to init microphone DOMException: Requested device not found initMic @ UseCarplayAudio.ts:78

AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. https://goo.gl/7K7WLu PcmPlayer @ PcmPlayer.ts:20

A few messages of:

delay jmuxer.min.js:1

It also seems that mouse/touch inputs are not working correctly or the video is delayed. I have tried to record it here: https://youtu.be/91Ji8C77TwQ You can hear me clicking with a mouse but the buttons in the navbar are not responding. When I drag the mouse over the map, it does seem to respond albeit delayed. Is this related to the new web integration/example somehow? I used to use react-carplay which would run in it's own AppImage, not much problems with delayed touch/inputs there.

All in all great that Android Auto seems to be working better though!

@steelbrain I have tried both wireless and wired on the CCPA, my android phones are not recognized at all using this dongle. iPhones do work. I'll try the other CCPA dongle that I have in my car later. After upgrading to a new firmware version (I assume through the web interface at 192.168.43.1), is it possible to downgrade as well?

rhysmorgan134 commented 11 months ago

@Tigo2000 how long did you run AA for? The jmuxer buffer overflows after quite some time, it takes longer the more RAM you have too.

Tigo2000 commented 11 months ago

@rhysmorgan134 When I tested and recorded that video earlier today, the projection was up for a few minutes. After your post I have kept the example running for just over an hour now and I don't notice any memory leaks or anything else. At the beginning the memory usage was at 1.12G / 3.71G, during the last hour it jumped around a bit, maxing out at 1.2G but usually hovering around 1.15G.

The video feed is still delayed and there's no audio.

N95JPL commented 11 months ago

When I was testing AA, before you wonderful people were involved, I needed to send a keyframe every so often to make the video catch up, I also had to change a setting on the box itself to use "Software" processing IIRC, that smoothed things out

This was ~6 months ago so going off long term, poor, memory here!

steelbrain commented 11 months ago

Failed to init microphone DOMException: Requested device not found initMic @ UseCarplayAudio.ts:78

AudioContext was not allowed to start. It must be resumed (or created) after a user gesture > on the page. https://goo.gl/7K7WLu PcmPlayer @ PcmPlayer.ts:20

Re audio not working, if you're on chrome, you may have to run it with --autoplay-policy=no-user-gesture-required to get rid of the second issue, for the first one, you can either make a dummy mic device on your system or plug something in temporarily to test.

Re video delay, it disappered (at least on Carplay) after https://github.com/rhysmorgan134/node-CarPlay/pull/34. It's interesting that it still exists on the android side. There's a changeset by fellow community members at Tesla-Carplay (https://github.com/marcraft2/tesla-carplay/pull/19) where they implemented a h264 to canvas rendering method where they just drop any overlapping frames (they did this in Canvas because Tesla stopped playing video when driving, jmuxer uses <video /> tag). It could be interesting to see if it'll help with video processing delay for Android.

Re CPC200-CCPA not working, I had a bumpy experience with the dongle (not just AA, but CarPlay as well) and managed to break the web interface through the integrated web-based upgrade feature. I contacted support and they sent me an update that can be flashed through a usb drive, worked for me, your mileage may vary. Let me know if you want it.

My current version for it is 2023.05.29.1924

Tigo2000 commented 11 months ago

Re audio not working, if you're on chrome, you may have to run it with --autoplay-policy=no-user-gesture-required

That seemed to fix that, thanks! The audio does skip every now and then, it seems like it's mostly when the CPU utilization is very high, when I move the screen @ 1:00 for example. I have tried to record it here: https://youtu.be/83E0dbr1Z6Q?t=25. The song is Fisher - Losing it with a pretty repetitive rhythm so it's easy to spot if there's any skipping. In the console, I can see a few messages delay from jmuxer.min.js

Is the audio quality on your end (using AA) the same as what you hear in that video?

Could you send me the new firmware for the CCPA? You can contact me on Discord at Tigo. Thanks!

gozmanyoni commented 11 months ago

@Tigo2000 Is this a Pi 3?

Edit: Saw it's a pi 4 in the first post.

Generally - you can reduce the config fps to reduce cpu usage and smooth things out, the example is at 60, could you try with 30?

Tigo2000 commented 11 months ago

@gozmanyoni This is indeed a Pi 4, 4 GB with a heatsink. I changed FPS to 30 and while audio playback is definitely better / less skippy (and cpu usage went down), the overall responsiveness of the UI just isn't there sadly.

I've also tried iOS/Carplay on the carplay-web-app example and that is also not very responsive. It takes quite a bit of time before the map moves when I drag the map with the mouse, as you can see in the uploaded videos.

Is it possible I'm just doing something wrong? I'm using the default/pre-installed chromium-browser on a fresh install of bullseye 64 bits with the experimental flag set, frameinterval set to null and fps set at 30. I've also changed resolution to 800x600 (or something along those lines). React-carplay or any electron implementation used to run fine, should I not compare this new version with that? Maybe my expectation is just wrong haha.

Anyway, I will wait for Rhys' new react-carplay version to rule out the new WebUSB implemention, as I assume this will just be another electron AppImage?

Thanks for all of you guys' hard work on this!

PS. I've also updated my dongle to same version as @steelbrain, sadly android devices are still not recognized.

rhysmorgan134 commented 11 months ago

@Tigo2000 good timing, super early version is up if you want to give it a test https://github.com/rhysmorgan134/react-carplay/releases/tag/v4.0.0-beta.1 to be honest it won't rule a huge amount out, the implementation is taken pretty much straight from the example here, just a bit of hackery to get the ring-buffer to work in electron, and few webusb bits. Loading needs the dongle plugged in then the button pressed, once loaded performance is flawless for me, so I would say if you have issues I would maybe look towards sd card/power. Are you running a vnc session on a headless pi by any chance?

gozmanyoni commented 11 months ago

@Tigo2000 did you have any luck with this version?

Also - the latest master has a more efficient rendering method - I recommend you give this a go

Tigo2000 commented 11 months ago

@gozmanyoni Yes I tried the beta.1, 2 and 3 on react-carplay, that seems to run much better than the carplay-web-app! It's smooth, audio playback is good and overall the experience is good. I used to have horizontal and vertical scrollbars on AA, but this has been semi-fixed by forcing the resolution to one of the resolutions that AA supports (800x480, 1280x720 and/or 1920x1080).

However after a few minutes it seems like touch/mouse click-inputs aren't registered/recognized anymore, but dragging the mouse to move the map (in Google Maps) around still works. After a little while (few seconds to half a minute) the click-input starts working again (so I can press buttons, navigate the menu, etc). I have not been able to reproduce when this is happening exactly as one moment it's working well and the next it's not recognized. It seems that when I pause or start playing music, the inputs start working right away again. It's almost as if the dongle went to sleep and it only wakes up after x amount of seconds or by an event like playing/pausing music.

Here's a brief summary of events:

I know it's pretty vague but do you recognize this behavior in a way that you can link it to the node-carplay implementation?

Just tested the newest node-carplay version as well and the implementation in the browser is still slow/laggy, but I take your word for it that it works better if this version would be in react-carplay for example. Nice job!

I also retrieved the other CCPA dongle from my car, this dongle also does not recognize any Android phones sadly.

rhysmorgan134 commented 11 months ago

I've been working with Tigo a bit on this, and last night I opened up the AutoKit app on an android device, even on this app it only lists the 3 possible resolutions. A bit rubbish of carlinkit since I believe Android have updated and now allow resolution negotiation from the car

https://support.google.com/androidauto/thread/8844203/android-auto-scale-on-car-screen?hl=en

unfortunately I think we will be tied to this unless there is a dongle firmware update to allow custom resolutions. Perhaps a feature of the CCPA? Maybe why it does not work with the default android settings. Purely speculation at this point.

gozmanyoni commented 11 months ago

Hmm - with touch being unresponsive - I have experienced this a few times, but never at a consistent pattern, and attributed it to issues in a dev env.

It would be one of two things - either we are not sending through touch events to the dongle (main thread blocked, or perhaps an issue with the driver) - and that's fixable on our side, Or the events are sent, and the dongle does not process them.

I will try to reproduce with carplay, but don't have an android device here to test.

With the frame Interval not running on android, perhaps it does leak memory with jmuxer, and as it slows down, the main thread develops delays and quirks - just a theory - the current implementation, in my short testing - does not increase on memory usage even with carplay when we dont have a frame interval set up - but i did only test for ~10 minutes.

rhysmorgan134 commented 11 months ago

@Tigo2000 the example plays by default at 60fps, which if you are on a 1080 monitor it is a bit much for the pi (even pi4 in my testing)

@gozmanyoni the webcodecs works great on my mac and running the example on the pi. I actually find jmuxer in electron to be smoother when on the pi though, unsure but pure speculation, perhaps the gpu is struggling with the load, as cpu usage is minimal in the new implementation.

My non scientific analysis is as follows

decoder resolution fps feel
Web Codecs 800 x 480 60 snappy smooth ~20% cpu
jmuxer 800 x 480 60 as above ~40% cpu
Web Codecs 1920 x 1080 60 glitchy delayed response ~ 50% cpu
jmuxer 1920 x 1080 60 unuseable ~90% cpu
Web Codecs 1920 x 1080 30 useable, but the odd lag ~ 40% cpu
jmuxer 1920 x 1080 30 perfectly smooth ~70% cpu

Worth noting the webcodecs can't be run in electron as electron pi builds don't support accellerated decoding

https://github.com/electron/electron/issues/34825

https://github.com/RPi-Distro/chromium-browser/issues/36

It's obviously only to do with the example so not an issue. Once I have finished the react-carplay v4 I'll add a trimmed down version for in the examples of this repo

Interested to see how the pi5 will handle it all!

gozmanyoni commented 11 months ago

@rhysmorgan134 from that thread it looks like it can be enabled during the build step though - I'll give that a go when I get a chance

rhysmorgan134 commented 11 months ago

@gozmanyoni Yeah it's been on my back log to try for a long time, but as I don't think it benefits jmuxer I never really progressed it. With web codecs I guess it's worth a try.

I have just found out apparently they have dropped h264 hardware acceleration on the pi5!

https://osmc.tv/2023/09/raspberry-pi-5-and-osmc-support-changes/

Apparently it can handle h264 decoding on a single core no problem. Trying to find if there is more info on this.

gozmanyoni commented 11 months ago

@rhysmorgan134 to break this down, I think the current state is the following:

Juicer produces a fragmented mp4 which the browser plays using hardware decoding from a video element, but adds the overhead of muxing, it's also all done on the main thread.

The WebCodecs based solution removes that extra load, and also offloads all decoding and handling off the main thread.

However, given that electron won't allow hardware decoding by default, it's using software decoding, and loses the benefits introduced.

Did you also try the example on a Pi, or only electron?

rhysmorgan134 commented 11 months ago

@gozmanyoni all rows that are listed on that table that use WebCodecs are run via the example, WebCodecs errors on and won't run. Jmuxer is run via electron. It certainly feels like a performance decrease in a direct comparison between WebCodecs (HW accelerated) and Jmuxer/electron (software decode) with Jmuxer outperforming, I will do some more testing tonight.

gozmanyoni commented 11 months ago

Ah, thanks for clarifying @rhysmorgan134 I will test on my side as well and try to get FPS counts, on my M1 Mac it feels snappier but I did not measure beyond.

How does your chrome://gpu page look?

rhysmorgan134 commented 11 months ago

A diff check of os chromium (left) electron (right)

https://www.diffchecker.com/soNXZPkZ/

gozmanyoni commented 8 months ago

@Tigo2000 Curious - do these still persist for you with latest builds and all of the optimisations? (especially since we made the changes to clear the main event loop from any touch interaction or event processing for video/audio)

Tigo2000 commented 8 months ago

Hey Yoni, no Android Auto is running pretty smoothly now! Haven't had any issues as of late. I think we can close this, unless you want to keep it open to troubleshoot wireless Android Auto. Steelbrain did get it to run on his end but I haven't been able to get wireless AA running/recognized yet.

If you think there's any improvement possible on node-carplay regarding wireless AA, it would be great if we could investigate further. If there's nothing more that we can do, I'm fine with closing this issue.

Thank you!

gozmanyoni commented 8 months ago

Thanks @Tigo2000! I will close this one then - but we can triage with @steelbrain , perhaps on slack and see if wireless AA has any issues