Surround sound

[mariotaku]

Moonlight is almost able to handle it, but hard to test yet.

[chocomega]

Interesting! @astarktc which playback device did you use in Windows to setup Dolby Atmos? I'm using Steam Streaming Speakers device and the Dolby Access app says it's not compatible.

[pankompot]

I'm in the same situation as @chocomega. When I connect laptop to AVR and do dolby atmos setup + moonlight on windows - it works perfectly fine but I cannot get atmos sound via webos app. Maybe some virtual sound device would be an option here?

[mariotaku]

No matter what setup the computer has, Sunshine (or GFE) transfers OPUS, which will not be affected by anything else.

I'll bring AC-3 audio encoding into discussion of Sunshine's development. But keep in mind that AC-3 has larger samples per frame (1536), which is 6.4 times of OPUS codec. It means you'll have at least 32ms of audio latency if we do have AC-3 audio, instead of the current 5ms.

[astarktc]

@chocomega @pankompot

In my case the playback device is NVIDIA High Definition Video / LG TV SCCR2:

So, much like HDR, it's entirely possible that without a TV / Receiver / Soundbar connected to HDMI out that the option may not be available except with a virtual device or headless hdmi adaptor that mimics Atmos support.

@mariotaku I don't know much about OPUS, but it sounds like you're saying that I should only be getting stereo audio? Let me know if I'm misunderstanding. Although I haven't streamed a game in the last couple weeks and haven't tested on my new receiver yet, I definitely remember getting discrete rear channels on Moonlight-tv despite the stereo setting in the moonlight menu.

[astarktc]

@mariotaku

Just another thought that crossed my mind, I realize maybe what you said about AC-3 might only apply to those who don't have the benefit of my specific setup, but fwiw I can't say I've experienced noticeable audio lag (not saying it doesn't exist, but audio is in sync with my screen and my video doesn't lag behind my controller inputs).

So, maybe another path forward in sunshine could be a virtual audio device that lets Windows handle the atmos encoding natively (which is going to be either dolby digital plus or TrueHD, neither are AC-3. AC-3 is 5.1 only, if the hypothetical lag was the same, I wouldn't limit it to AC-3) and pass it through to the moonlight client?

I don't know the first thing about the efficiency of encoding DD+ or TrueHD vs AC-3, but, there are games that natively support atmos output as an option in the game menu, so there has to be some kind of existing API or maybe the format was built to encode the audio efficiently enough to avoid introducing lag?

I wish I was a programmer, I would help build these features if I could :(

[pankompot]

@astarktc it's already interesting that you see your TV in devices when using sunshine+moonlight. Could you describe your setup in a bit more detail? Looking at your screenshot it seems like your TV would be connected to host pc directly with hdmi. This is not the case right?

[astarktc]

@pankompot you're right, that's not the case...at least, not in the way you might be thinking :) I should have described my setup in more detail, so here it goes:

The TVs that I stream to are an LG C2 65" in my bedroom and an LG C1 83" in my living room. My gaming PC is in my home office, and it's directly connected with HDMI to an LG C2 48" as the monitor, which I use when I game at the desk. Also, for sound, rather than use built-in TV audio I bought a Sonos Beam Gen 2 soundbar which does support receiving atmos sound input. I'm not sure whether or not the soundbar specifically made a different, I built this set up last summer just before discovering game streaming.

So, that's why you see a TV in my devices: that's the monitor in my office. But I do 90% of my PC gaming via moonlight so I can lounge in bed or on the couch. The desk is just for productivity at this point lol.

Back to the client side, the bedroom TV is connected to a Samsung soundbar with atmos support via up-firing speakers, the living room is my home theater & gaming room which had an older Denon AVR-S920W (that I was running in 7.1) which I just replaced 2 weeks ago with a Denon AVR-X3800H running the same speakers plus 4 new in-ceiling speakers for Atmos. So far, I've only tested movies with the new receiver.

Besides all of that, my host PC runs an RTX 4090 and Core i9-13900K

[pankompot]

Thanks for all the details and kudos for sweet setup ;) This is all really interesting, I'll try to play with some virtual audio or just get a Atmos capable soundbar to my home office, perhaps it's high time for this anyway! @mariotaku what do you think about this considering OPUS limits you mentioned? If Atmos sound couldn't get through how would AVR recognize it?

[astarktc]

No problem and thanks :) so, just confirm I'm not crazy, I'll try to make time today to stream a game in the living room. If the receiver displays atmos on the front, I'll know I'm not crazy or delusional ;) The only verifiable fact I do have is that I heard discrete sounds behind me, in-game.

If I knew fully how the internal mechanics work, I'm sure that would explain why/how this might be working. I can only theorize. @mariotaku is way more knowledgeable and experienced enough to decipher what's going on.

My theory, based on the atmos support in Windows, Xbox, and PS5, there's certainly something that sunshine might be able to hook into without having to take 5.1 or 7.1 LPCM and encode it. I don't know if the games are passing raw spatial audio data into an API or if the games are internally encoding Dolby before passing it on to the audio device.

But let's think about it this way: if the audio was being encoded before it reaches sunshine, it would actually introduce more lag to decode and re-encode into OPUS, AC-3, or any codec. It's also possible that by running atmos for home theater mode on my pc, that games are at least generating 5.1 or 7.1 lpcm that sunshine is able to encode and pass on, but since we know that this is disabled in the official webos release, I'm inclined to believe that somehow, sunshine was either smart enough to passthrough the encoded audio or just accidently did it because it didn't know how to interpret it. at that point, for all I know, the decoder might have recognized my LG's passthrough setting and skipped audio decode and passed it on. either way, a surround track managed to reach my receiver haha.

If it doesn't work when I test it today, needless to say I'll be embarassed ;)

Edit: since it's likely relevant, my moonlight client is just using the default decoder settings, with HDR & hevc enabled

[marvin-roesch]

That's very interesting, @astarktc. I'll have to try out using the Dolby Atmos functionality in Windows myself and will report back.

From a purely theoretical standpoint, it'd be quite astonishing if this works. As @mariotaku pointed out, Sunshine encodes the raw audio streams it captures from the device for each channel. The Moonlight TV app in turn passes the raw Opus signal onto the TV which decodes them back into PCM data. So if the Dolby Atmos setup somehow implants itself in the audio device streams such that the data Sunshine gets is somehow the digital Atmos signal, that would still have to survive the lossy Opus encoding step, which I find highly unlikely.

FWIW, @mariotaku, I have experimented a tiny amount with getting AAC (not AC3) encoding into Sunshine and passing that onto the TV/soundbar with moonlight-tv, since that should get passed through as is. The encoding went reasonably well, but unfortunately I only got out garbage when trying to play it back on the TV, but I'm also not at all knowledgeable with the LG SDKs.

I have since gotten an NVIDIA Shield to take care of my game streaming needs (including surround sound), but I'm still open to helping out wherever I can.

[astarktc]

Ok, I did a test, and things are a little weird...

First thing, my LG's dev mode had reset, so I had to redo that process and re-install moonlight. I had been busy and hadn't played a game for 6+ weeks. Things aren't working like they used to, but the way this is manifesting has me questioning if an update (whether moonlight, webos, sunshine, etc) has broken my workaround or if the old receiver was upmixing the whole time.

I tried two games, one of them just plays stereo, and the other isn't playing sound at all (the one with advertised atmos support). It's possible my old receiver was in an upmixing mode, but I don't think upmixing is capable of knowing where an object is in 3d game space and transitioning it from in front of me, to behind my right shoulder, left behind my left as I rotate the camera. That's something that used to happen, where the sound was coming out of that speaker exclusively.

But the real reason I'm wondering if an update broke it is because the 2nd game i tested was plaque tale requiem, which worked perfectly fine in the past. Now, it only played sound (stereo) during the pre- menu intro video, and now it's silent in the menu and game. When I played through the game late last year, it was entirely through moonlight.

Nothing has changed between now and then except software updates for moonlight, the TV, sunshine, GPU driver, and the new receiver. You can see in the picture that it's in home theater mode with 3d audio on.

The key fact for me is: its silence tells me the spatial audio is blocking the sunshine from capturing audio. This didn't used to be the case. I'm fairly confident of this, but only way I could prove myself wrong is to roll back the software versions of everything which is too impractical.

So maybe we're back to square one, so I apologize if I got anyone's hopes up.

But @mariotaku , what do you think about the fact that I'm not getting audio at all from plaque tale? It seems to indicate that the normal methods of capturing audio are unavailable in this mode. If I quit game, turn off spatial audio, and open it I hear the game again.

Rather than building an AC-3 encoder, is it possible to configure sunshine to capture the dolby encoded audio stream and pass it to moonlight, which can just pass it through eARC to the receiver just like Netflix and other apps do? It sounds like such an elegant solution to the problem and I think it will avoid the additional lag you mentioned, as we'd be hooking into a native solution that works lag-free already.

It's like Microsoft handed us the solution on a silver platter, and the only thing that might get in our way is LG.

[mariotaku]

@astarktc @pankompot I'll try to explain how Sunshine on Windows transfers audio to Moonlight.

1. Sunshine captures PCM audio with IAudioCaptureClient
2. Sunshine encodes PCM into OPUS (lossy encoding)
3. Sunshine transfers OPUS frames to Moonlight
4. Moonlight decodes OPUS back to PCM
5. Moonlight sends PCM to OS for playback

So I doubt Dolby audio can be transferred at all. As for capturing Dolby audio stream - I'm afraid even we can do that, webOS will not be able to handle that either. It only supports PCM, AC3 on webOS 3 and 4. And PCM, OPUS on webOS 5+.

Guess this can't be verified without me having a real surround sound setup...

[mariotaku]

I have purchased a small gear to decode 5.1ch AC-3 from HDMI ARC. After it's arrival I'll do more research to see if it's possible to play surround sound.

[astarktc]

@mariotaku

Awesome, hopefully your research and testing pans out. I really appreciate your effort on this (and I'm sure many others do too!).

BTW, I figured out that the DD+ codec is EAC3, which is the most likely format that this windows atmos for home theater is using for output. Hopefully, the device you bought supports that too, but if it doesn't work for what you're trying to test, if it's AC-3 only that could be a reason why.

Don't know if you found these already, but I found a few developer focused resources in case they're helpful:

https://professionalsupport.dolby.com/s/article/Dolby-Atmos-ADM-Profile-specification?language=en_US

https://developer.dolby.com/platforms/windows/dolby-atmos/implementation/

https://learn.microsoft.com/en-us/windows/win32/coreaudio/spatial-sound?redirectedfrom=MSDN

Now, in regards to the formats you mentioned that webos can handle, that's for decoding right?

From what I'm reading on LG's dev site, those are the formats supported for standalone audio. If sunshine is streaming the video and audio separately, that makes sense. The key to supporting AC3, EAC3, etc is to mux the audio and video together into a single stream via the mkv, mp4, or ts formats:

https://webostv.developer.lge.com/develop/specifications/video-audio-230

So, I think what you're looking to implement is what software like Plex, Kodi etc do: stream a muxed audio/video format to the TV over the network.

If in your research you find folks complaining about plex not streaming atmos on the webos app, I now realized based on that info above that it's because those folks are usually trying to play truehd audio from blurays, and that documentation makes it plain that truehd isn't a supported format, but DD+ (EAC3) is supported, so that won't prevent you from accomplishing the goal here.

Technically, I don't even want moonlightv to decode the audio at all, I want it to pass through to the receiver. However, I think once the stream is streaming through mkv, mp4, or ts, that the webos decoder is automatically going to follow the tv's settings and pass the EAC3 through to the receiver.

Just thinking outside the box, one other area we can also use to look for this information might be their DRM documentation. We all know the content creators/studios pushed hard for content protection for media played through streaming apps like Netflix, so maybe we'll find what you need here?

https://webostv.developer.lge.com/develop/specifications/streaming-protocol-drm

https://webostv.developer.lge.com/develop/guides/drm-content-playback

[mariotaku]

@astarktc There are multiple principles for surround sound codec:

• It has to be real time streaming capable and as low-latency as possible. Thus muxing video and audio won't be an option.
• It must not have any copyright/patent issue, so something like Dolby Atmos won't be possible to be implemented by sunshine or moonlight at all.
• It has to be supported by HDMI ARC (and eARC) standard

That's why AC-3 is the only option left. It can be encoded by FFMPEG, seems to be supported by webOS when streaming audio. Also for newer models, it looks like they support 6-channel PCM too so there's hope.

[astarktc]

I understand. We have to work within the limits we have, unfortunately. It would be nice if LG didn't make this difficult, especially in 2024.

I did some more research, and copyright wouldn't be an issue for atmos as that is there is no license cost for atmos for games through windows.

Truly, our only roadblock is the inability to mux without introducing latency 😞 if standalone e-ac3 decoding was supported in webos, it would be easy to capture the encoded audio from windows spatial sound and pass it through, even if we were only targeting 5.1/7.1 and discarding the atmos metadata.

What I need to find is the least expensive client device that can support HDR, 7.1, and 120fps at the same time. Probably a mini or sff pc. That seems to be the only way to avoid any compromises.

[mariotaku]

OK so here comes my first update of surround sound feature take 2:

Proof of concept works! OPUS audio got decoded to PCM, re-encoded to AAC-LC on TV and passthrough to the receiver. I can hear sound coming from the correct speaker.

However there are more problems that need to be solved:

1. Latency is over 1000ms, which is obviously unacceptable
2. Encoding audio shouldn't happen on TV, 1024 audio samples (21.3ms) needs 25ms to encode
3. License issue. While FFMPEG can encode AAC, if the GPL licensed encoder doesn't work then it would be difficult to add another audio codec for Sunshine.

[marvin-roesch]

@mariotaku While I've moved onto streaming via NVIDIA Shield, I've still got my modifications to Sunshine for experimentally switching the audio codec to AAC-LC, encoded using the Fraunhofer FDK AAC.

I'll see how easy that is to update to the latest Sunshine version and then maybe push the fork so you can try it out for yourself to check if moving the encoding step right to Sunshine helps or if the encoder simply isn't fast enough.

[mariotaku]

Good news for everyone - 5.1 surround is really working & with pretty good performance!

There are a few limitations:

• This only works on webOS 5+ at this moment, maybe the same tech can be applied to webOS 4, but I have no plan supporting surround on even older platforms.
• webOS can only recognize a certain audio config, so audio stream has to be encoded again on the TV which is not ideal. After https://github.com/LizardByte/Sunshine/pull/2424 gets merged & released, double encoding will no longer needed.

[GingerAdonis]

Great to hear. Very exciting. Is a new release on your side required as well?

A backport to webOS 4/4.5 would be appreciated.

[mariotaku]

@GingerAdonis Hi, I also tried to work on webOS 4, but unfortunately, I didn't find a way to play surround audio with low latency yet (it has more than 1000ms of latency). So I'm unable to support webOS 4 and below at this moment.

[GingerAdonis]

@GingerAdonis Hi, I also tried to work on webOS 4, but unfortunately, I didn't find a way to play surround audio with low latency yet (it has more than 1000ms of latency). So I'm unable to support webOS 4 and below at this moment.

Thank you for giving a shot. :)

[mariotaku]

This feature has been released in 1.6.13 and above. Finally it's implemented and thanks everyone for testing & feedback & providing information!

[drsno]

Great work! One thing that is not entirely clear yet, does this also work with GFE or only with Sunshine?

[mariotaku]

Great work! One thing that is not entirely clear yet, does this also work with GFE or only with Sunshine?

It should work with GFE and the current version of Sunshine, with a minor performance degrade. Once my patch for Sunshine is released, the performance will be even better (5ms less latency)

[Maltique]

Should this work out of the box ?

I installed 1.6.15, and sunshine's current version (I wanted to test it even though the latency is not fixed yet in it).

But, it only shows up as STEREO PCM on my sound bar, even though I set the audio to passthrough on my TV.

I even tried to install the steam speaker, and set it to 7.1 or 5.1 without success so far.

Thanks a lot for the continued work. I am happy to see such feature arriving to this beautiful port 🙂

[mariotaku]

@Maltique It supposed to work on webOS 5+ if you select "5.1 surround". Can you provide more info?

[Maltique]

When using GFE, I actually get Multichannel PCM (Yay :D).

Somehow, when using sunshine it does not work. I may try to reinstall my sunshine or look at tips on the internet.

[mariotaku]

When using GFE, I actually get Multichannel PCM (Yay :D).

Somehow, when using sunshine it does not work. I may try to reinstall my sunshine or look at tips on the internet.

Got it. For future discussions about this issue you see, please create a new one, so we can drop more info there.

[Maltique]

My bad here, GFE is always in surround mode whereas Sunshine switches to it only in games. After some more digging I have been able to make it work.

Thanks @mariotaku

[gotwig]

@Maltique It supposed to work on webOS 5+ if you select "5.1 surround". Can you provide more info?

So it doesnt work on WebOS 4? I have a LG C9. I can select the option in settings but had no chance to try yet

[mariotaku]

@gotwig No, it doesn't work on webOS 4 and I'm unable to find out any method to achieve that.

[astarktc]

@mariotaku I wanted to take a moment to give you my thanks, gratitude, and kudos for fixing surround sound on WebOS 😊

I tested it, and it works perfectly. Latency is imperceptible. With a true 5.1 signal, I feel better about using the upmixing mode on my receiver for the 7.1 effect 😉

[Calprainn]

I would also like to say a big thank you to @mariotaku for this really excellent work!! 😊👍 I've been waiting for this for a long time and now I can finally play my games on the LG in 4K, HDR, 120fps and 5.1 surround sound.

[rayfilm0815]

@astarktc I would like to ask if you are using a soundbar? Because I am currently encountering the same delay problem that I have encountered before.

When the TV is set to pass the signal to the soundbar, the surround sound works normally but there is a delay issue If set to PCM, I can't get surround sound because LG doesn't seem to support surround sound PCM. It’s amazing if you can still get lag-free performance when using a soundbar!

[astarktc]

@rayfilm0815

Half the time I am, half the time I'm not. My LG in the living room has a denon receiver, and the one in my bedroom has a Samsung soundbar. No delay on either 🙂

Is this only happening for you on moonlight, or other content too?

One thing you could try is under LG's sound settings -> advanced -> match screen and sound, and try turning on 'Bypass' if it isn't turned on already.