Offload all dongle data procesing from the main thread

[gozmanyoni]

To partially address https://github.com/rhysmorgan134/node-CarPlay/issues/37#issuecomment-1802456537 and come up with a pattern to do this I tackled offloading the video frames off the main thread using a MessageChannel.

What we have remaining is audio:

1. Playing audio which should probably done by sending the SharedArrayBuffer over to the carplay worker (together with the audio type), and write to it there instead of on the main thread - we will need to do this for every player created, so the carplay worked needs to be extended with new message types - one to request a player and one to receive and set one. the player has to be created on the main thread as you cannot initialize an audio context in web workers.
2. Mic input - can probably use a MessageChannel as well and skip the main thread completely.

[rhysmorgan134]

@gozmanyoni I haven given it a quick test on a pi and I'm presented with the below error. I haven't had a chance to investigate, I will have a further look tomorrow!

[gozmanyoni]

@rhysmorgan134 updated it to make sure we only transfer ownership of the channel to the worker once

[rhysmorgan134]

Seems to get further, so far I have only tested on a pi5, but now errors here, could well be pi5 specific, so will test on a pi4 when I can

[gozmanyoni]

@rhysmorgan134 hmm, this seems to be related to the decoder not supporting the format - curious to see the gpu page on chromium on the Pi 5.

I haven't received my unit yet so cant test on a pi 5 :(

[gozmanyoni]

@rhysmorgan134 @steelbrain ok - I managed to get both microphone and audio players streaming directly off their own respective threads with the main thread only orchestrating things at this point.

Still needs a refactor but good for testing. works well on my mac, will try on a pi 4 as well, no pi 5 yet

[rhysmorgan134]

@gozmanyoni Amazing stuff! I'll give it a test.

I think the issue is that pi5 doesn't have hardware decoding for h264, so I guess it needs to revert back to software decoding when this happens, gpu page for a pi5 below

I know android auto can be output using h265, I will take a look back through the apk and see if it's possible with the carlink. I have always wondered what the format option is when configuring the dongle

[rhysmorgan134]

[gozmanyoni]

@rhysmorgan134 the decoder is configured with "prefer-hardware" I assumed it will fall back to software if hardware decoding is unsupported 😅 you can change that value in the render worker to software

[gozmanyoni]

Also, I tested various numbers in the format fields and couldn't get it to have an impact on the codec. IOS also supports h265 CarPlay, but I'm not sure how to make the dongle stream that or if it can.

[gozmanyoni]

Ok, tested this against master on my pi 4, I have a 9.3" 1600x600 touch screen for my project. Previous version would log 30-49 fps.

New version feels snappier and is much more stable at 55fps

[rhysmorgan134]

@gozmanyoni Working well with prefer-software on the pi5! How are you benchmarking fps? Using the chrome fps utility mines reporting a solid 52.2 on a 1080 screen, doesn't seem to ever budge, so I suspect that might be a pseudo figure. Strangely with it set to prefer-software, the gpu usage is pretty maxed out

[rhysmorgan134]

@gozmanyoni As an side I have got piMost fully working on the pi5, these two issues were the cause, just need to pull a device agnostic fix into the repos

https://github.com/fivdi/onoff/issues/197

https://forums.raspberrypi.com/viewtopic.php?p=2156043#p2156043

[gozmanyoni]

@rhysmorgan134 the renderer can report fps in the console, if you initialise it this way :

worker.postMessage(new InitEvent(canvas, videoChannel.port2, undefined, true),

true/false is fps reporting. it will print the fps in the video every 5s.

Great news about it working on the Pi 5! excited to try once I finally get mine. With these changes though the Pi 4 is doing quite well - 55fps is nearly as good as it gets!

Excited for Pi 5 fast boot times, will be really great to boot it off a super fast NVME and a light OS into chromium to run the carplay react app.

For GPU usage - even if you are not using hardware decoding for h264 its using WebGL to render the decoded frames - thats the GPU usage

[rhysmorgan134]

@gozmanyoni I report various values from 51 to 38, however I am unsure how accurate that is, as the dongle only sends video packets when the image has changed, you can tell this by leaving it on the home page and you only receive a packet at the key frame interval rate or if the minute on the time changes. It feels mega smooth so I would say when it's needed it is reaching the 60.

My pi5 with an sd card boots to the desktop in 10 seconds, vs pi4 at around the 24 seconds mark. With a pcie ssd I can image it is going to be even faster. There's rumours that with the new Rp1 io chip, it may even finally support a power effecient suspend, which would be brilliant for an automotive set up.

edit - also if you have ordered one, make sure you grab the cooler, and also the official case. The piMost fits in the official case almost perfectly, and the cooler is the ideal height

[rhysmorgan134]

[gozmanyoni]

@rhysmorgan134 finally managed to get a Pi 5! CarPlay running super smooth on it!

Do you have a branch of SocketMost that will work on it by any chance? :-)

[rhysmorgan134]

@gozmanyoni latest push should be the one! However I believe the nom version should already support it

[gozmanyoni]

@rhysmorgan134 gave it a try from the typescript branch which has the pi5 changes - no luck so far.

Read me needs an update since now it seems that first we need to run build, and set up the socketmost service to run examples/server.js

But it faults and the PiMost never lights up for me

[rhysmorgan134]

@gozmanyoni can you try the latest push to typescript? I had execSync all wrong to detect whether it's a pi5 or 4. But yeah npm run build, then examples, node server.js

The red light will only come on when there's activity on the rx light

[rhysmorgan134]

@gozmanyoni I have also uploaded the latest most-explorer app, only done the mac binary at the moment, will do the rest later today. Just needs server.js started in socket most on the pi, and also explorerServer.js.

I need a big tidy up! Node-carplay has been the main focus.

[gozmanyoni]

Logs look better with last pull! :-) Will try with the pi in The car tomorrow!

The old PiMost would light up when the pi booted - might be related to the previous SocketMost implementation.

[rhysmorgan134]

Ah yeah that was an improvement to it a while ago, it only enables the output when an input is present, otherwise other modules don't like a light when the master has turned off.

[gozmanyoni]

Let me know what bits you plan to clean up, happy to pick up any cleanup tasks or refactor areas!

[rhysmorgan134]

@gozmanyoni Thanks Yoni, I think at the moment it's mainly just admin tidying, as I have bunch of uncommitted development work. I think I am there now, I have updated socketmost to include the dtoverlay for the pi5 too. Rest of the set up is the same as a pi4.

I have been working on a mostbus app for the JLR family, you may find some of the MOST implementations interesting. https://github.com/rhysmorgan134/JLR-HU/tree/main the interesting stuff is inside src/main/PiMostFunctions

It's super rough on the visuals at the moment, but I am quite happy with the mostbus comms, the AudioDiskPlayer may also mostly work with BMW as it uses standard blocks. AmFmTuner is completely custom though.