search

google / ExoPlayer

An extensible media player for Android
Apache License 2.0
21.65k stars 6k forks source link

opus 5.1 and 7.1 channel order is incorrect #8396

Closed Tupsi closed 7 months ago

Tupsi commented 3 years ago

Describe how the issue can be reproduced, ideally using the ExoPlayer demo app or a small sample app that you’re able to share as source code on GitHub. To increase the chance of your issue getting attention, please also include:

Observed behavior: Playback works but channels are somehow mixed up. The dialogs which should be on center are now ONLY on the right channel and the center only plays back some surround channels ambient music/sound.

Expected behavior: Channels should be where there are suppose to be. I can force that by enabling transcoding in the emby app (server side), so the server transcodes the audio from opus to dd+, resulting the dialog gets played at the center speaker).

You will NOT hear the problem with 2.0 stereo sets as the downmixing seems to work in the sense, that the original center gets mixed into both channels.

I also isolated only the center channel on my computer from the audio to verify that this is indeed the original center audio.

For reference, I already talked about this problem in detail over at the emby forum over a year ago and provided a sample file (in case you need one) https://emby.media/community/index.php?/topic/71360-playing-back-opus-5171-does-not-work-center-channel-is-missing/

marcbaechinger commented 3 years ago

Thanks! Can you provide us with a sample stream? I know you said it's in the thread you are referencing, but besides the thread being huge, I seem to not have access to the sample.zip that might be what you say is the sample stream.

Tupsi commented 3 years ago

Yeah sorry, just now realized I password protected that file because of the overzealous german laws when it comes to copyright stuff. Does github offer any form of private upload?

marcbaechinger commented 3 years ago

Please send an email to dev.exoplayer@gmail.com using a subject in the format "Issue #8396". Please also update this issue to indicate you’ve done this.

Many thanks!

Tupsi commented 3 years ago

Please send an email to dev.exoplayer@gmail.com using a subject in the format "Issue #8396". Please also update this issue to indicate you’ve done this.

Many thanks!

done.

krocard commented 3 years ago

I though at first that your channels where in a different order than what Android was expected. It does not seem to be the case as ffmpeg and android have the same channel order. I also looked at the sample and it has the expected channel order.

I then thought that maybe the format was played in passthrough and the tv/av receiver was decoding this incorrectly, but it unlikely that your setup supports Opus passthrough.

The channels also seems to be correctly mapped post decoding as dd+ is correctly played.

So the audio is most probably decoded on your FireTV Stick, and the channel incorrectly mapped there.

Unfortunately, I don't have access to a FireTV, nor a 5.1 setup.

In order to isolate variables and confirm the issue, could you play your sample with our DemoApplication? If our demo app also exhibits the issue, then it is most probably a platform decoder issue, and you could block this decoder, if there is an other one on the platform supporting Opus or use the ffmpeg extension. If our demo app does not exhibit the issue, then something else in your setup is causing the problem and I don't think we can help you figuring out what it is.

Tupsi commented 3 years ago

Sorry, but I am not enough of a programmer to get that DemoApplication running. While I could try to install Android Studio there is talk about "connecting a physical device" and I have no clue how to do that and what is actually ment by that.

At least I can give some more information to convince you that the bug is actually in the exoplayer code and not somewhere else. I installed the Kodi app (which to my knowledge does not use any of your code) on FireTV app and it plays my files nicely, meaning opus audio always has the correct channel order. Over the holidays, I switched to Nvidia Shield and installed Emby there with the same result, channels are not where they should be. The emby people tell me that for playback of stuff like opus, they use your code and do not have anything else for it, hence my cry for help here.

You mentioning to "block the decoder" would not really help me, as emby rely on your code to decode it, but maybe I do not understand really what you are trying to say with that.

Tupsi commented 3 years ago

ah wth, I will give it a try, but its to late for today, so I will try tomorrow. Can you give me some pointers please in how to connect that compiled app to my shield and how I would have to edit that playlist I seem to need?

krocard commented 3 years ago

Sorry, but I am not enough of a programmer to get that DemoApplication running. While I could try to install Android Studio there is talk about "connecting a physical device" and I have no clue how to do that and what is actually ment by that.

I will take a look if we have a prebuilt DemoApp for you to install direcly instead of building your own.

I installed the Kodi app (which to my knowledge does not use any of your code) on FireTV app and it plays my files nicely, meaning opus audio always has the correct channel order.

I had a quick look to the Kodi and it seems to be using ffmpeg, not the platform decoders.

The emby people tell me that for playback of stuff like opus, they use your code and do not have anything else for it, hence my cry for help here.

ExoPlayer does not decode itself media, it uses the platform decoders (aka the Opus decoder bundled by Amazon in the FireTV os), and optionally ffmpeg if bundled.

If that platform opus decoder has a channel mapping bug, your best way to avoid the issue would be to ask the emby app to use the ExoPlayer extension to offer the option to use ffmpeg instead of the platform decoder.

But first we need to make sure that the platform Opus decoder is indeed the problem and not some other code. Let me search for an app that offers both ffmpeg and platform decoding to compare both decoding.

krocard commented 3 years ago

Can you try playing your sample with VLC for Android and compare that to playing the sample with the galerie app of your FireTV? If the galerie app reproduces the issue, it will definitely be a platform issue.

Tupsi commented 3 years ago

Can you try playing your sample with VLC for Android and compare that to playing the sample with the galerie app of your FireTV? If the galerie app reproduces the issue, it will definitely be a platform issue.

Just installed VLC on the Shield and it plays all files I have in 5.1 or 7.1 opus as it should be (aka fine), not just the sample I send you. I could plug in the FireTV again as well, but as both devices show the same outcome, I see no point atm.

What exactly do you mean by "galerie app of your FireTV"?

From what you last replied it seemed that in both Amazon FireTV and Nvidia Shield your mentioned platform decoder gets used, which in turn means, you are unable to fix it in ExoPlayer itself because you do not handle the actual decoding. Which means I knocked on the wrong door to fix my problem. At least you offered a way out of this mess for me, IF I can convince the emby programmer to add that ExoPlayer extension.

In a general sense you guys might think about not using that plattform stuff if its that crappy and your have that ffmpeg option anyway in your product.

krocard commented 3 years ago

In a general sense you guys might think about not using that plattform stuff if its that crappy and your have that ffmpeg option anyway in your product.

Platform decoders are more power efficient, can handle higher bitrate/resolution and do not impact app size.

I'm surprised Nvidia Shield has the same issue. Could you try to play your file with the built-in media player? I'm not sure how to do it with the shield, but you should be able to download the file and then play it from the android file manager at least.

krocard commented 3 years ago

That would confirm it is indeed a platform decoder and not a software one.

Tupsi commented 3 years ago

Could you try to play your file with the built-in media player? I'm not sure how to do it with the shield, but you should be able to download the file and then play it from the android file manager at least.

confirmed. Playback with a file manager has the symptoms with switched channels.

Tupsi commented 3 years ago

That would confirm it is indeed a platform decoder and not a software one.

and that would mean my pilgrimage has to move on to where exactly? Nvidia Shield uses AndroidTV so I would assume google somewhere?

krocard commented 3 years ago

Thanks for testing @Tupsi. Now that we know it is a platform decoder issue, we need to know which decoder is used by the Nvidia Shield and which company is providing it. If it is a Google decoder, we will report it to the appropriate Google team to be fixed in the next Android release.

For that, we are going to need need a bug report that you should generate during a playback of the problematic media with the file manager or shortly after. See here on how to take a bug report.

krocard commented 3 years ago

@Tupsi I was able to reproduce on my phone tip of tree (dumping the 6 channels before they were downmixed to stereo). No need for you to provide a bug report. So it is indeed an AOSP issue, all android devices are probably affected. I will report it to the Google internal team in charge. Thanks for your persistence in reporting this issue. Internal issue: b/177524746.

krocard commented 3 years ago

As far as workaround, we already have a class that remaps channels, so we could imagine remaping the channels when this decoder is used with a multichannel output. Though it will depend if this channel issue is always present or just for some specifically constructed Opus file.

Tupsi commented 3 years ago

As far as workaround, we already have a class that remaps channels, so we could imagine remaping the channels when this decoder is used with a multichannel output. Though it will depend if this channel issue is always present or just for some specifically constructed Opus file.

well I do not know if my way of getting myself opus into my mkv is specific or rather general, but I will just post you my usual ffmpeg audio part here for reference:

5.1 -c:a libopus -b:a 192000 -mapping_family 1 -af aformat=channel_layouts=5.1 7.1 -c:a libopus -b:a 450000 -mapping_family 1 -af aformat=channel_layouts=7.1

so I do not think, that is anything special.

Tupsi commented 3 years ago

@Tupsi I was able to reproduce on my phone tip of tree (dumping the 6 channels before they were downmixed to stereo). No need for you to provide a bug report. So it is indeed an AOSP issue, all android devices are probably affected. I will report it to the Google internal team in charge. Thanks for your persistence in reporting this issue. Internal issue: b/177524746.

nice to see that we are getting somewhere and thanks for helping out!

Will you update this when there are changes to the base or is there some other place I could try to keep track, so I can revisit this once there is progress?

andrewlewis commented 3 years ago

With the ExoPlayer opus extension (not depending on the platform opus decoder) the same issue is reproducible.

I noticed in the ffmpeg sources that a different channel mapping is shown for 'vorbis' and 'libopus': https://github.com/FFmpeg/FFmpeg/blob/f4bdedd/libavcodec/libopusenc.c#L62-L84 If I step through the ExoPlayer opus decoder, we read a mapping for this file of 0 4 1 2 3 5, which appears in ffmpeg's vorbis mapping, and if I hardcode this to 0 1 4 2 3 5, which appears in ffmpeg's libopus mapping, the audio sounds correct.

krocard commented 3 years ago

well I do not know if my way of getting myself opus into my mkv is specific or rather general, but I will just post you my usual ffmpeg audio part here for reference.

I will update this issue when significant progress is made internally (b/177524746). In case I forget, you may request an update in a couple of month if nothing is posted here.

Tupsi commented 3 years ago

Thanks!

I found a DTS 7.1 Testaudio file today in my lib which nicely shows which speaker is currently suppose todo something in the test (in the video). So as expected this works as it should, my reciever gets the DTS Signal and decodes all channels properly, Then I put a 2nd audio stream in the mkv as 7.1 opus and now I get a better idea which channels are switched and to where, so I though I would share that with you guys. Maybe that helps hunting it down faster.


correct (DTS)       | wrong (opus)
----------------------------------------------------------
Left                | Left
Center              | Right
Right               | Center
Right Side Surround | Left Rear Surround
Right Rear Surround | Left Side Surround
Left Rear Surround  | Right Rear Surround
Left Side Surround  | LFE
LFE                 | Right Side Surround

with the left side "correct (DTS)" meaning that it works as intended and the right side "wrong(opus)" meaning that this is where I can hear the channels went.

Getting media info of the files show clearly that opus put the channels in a different order then DTS. 

Audio #1
ID                                       : 2
Format                                   : DTS XLL
Format/Info                              : Digital Theater Systems
Commercial name                          : DTS-HD Master Audio
Codec ID                                 : A_DTS
Duration                                 : 1 min 17 s
Bit rate mode                            : Variable
Bit rate                                 : 5 002 kb/s
Channel(s)                               : 8 channels
Channel layout                           : C L R LFE Lb Rb Lss Rss
Sampling rate                            : 48.0 kHz
Frame rate                               : 93.750 FPS (512 SPF)
Bit depth                                : 24 bits
Compression mode                         : Lossless
Stream size                              : 46.5 MiB (15%)
Title                                    : DTS-HD Master (7.1)
Language                                 : English
Default                                  : Yes
Forced                                   : No

Audio #2
ID                                       : 3
Format                                   : Opus
Codec ID                                 : A_OPUS
Duration                                 : 1 min 17 s
Bit rate                                 : 408 kb/s
Channel(s)                               : 8 channels
Channel layout                           : L R C Ls Rs Lb Rb LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 50.007 FPS (960 SPF)
Compression mode                         : Lossy
Stream size                              : 3.80 MiB (1%)
Title                                    : OPUS 7.1
Writing library                          : Lavc58.115.102 libopus
Language                                 : English
Default                                  : No
Forced                                   : No
Sray69 commented 2 years ago

I was wondering if there is any update on this issue? I can confirm that this is still an issue with the new Fire TV Stick 4K Max and ExoPlayer 2.17.1.

There was mention above of a possible workaround. Using ffmpeg? Re-encoding/Re-mapping channels? ExoPlayer plays AV1 files better than any other player on the Fire Stick. I really hope it gets worked out.

tonihei commented 2 years ago

There is no progress on the underlying platform bug I'm afraid.

The workaround mentioned above is to use a ChannelMappingAudioProcessor (that can be injected into the DefaultAudioSink returned from DefaultRenderersFactory.buildAudioSink). This processor can reorder channels as required. However, you'd need to know exactly when the problem will occur in advance to be able to add the processor to the player only when needed.

Sray69 commented 2 years ago

Thanks for the response. I ended up finding a player that actually works.

Tupsi commented 2 years ago

My problem occured in the software emby. After convincing the guys finally with this that there is actually a problem in the ExePlayer code (which they use underneath) they just switched to software decode and let them handle it (guess with ffmpeg) and now everything is fine. Hope you guys get it fixed someday, I have my fingers crossed! After all, it was a nightmare convincing the guys that I am not a total idiot in encoding stuff, but that there was a problem with the decoder. :-)

Sray69 commented 2 years ago

Thanks for that info. I don't have much confidence it will get fixed any time soon. Which is odd to me because I would think it affects a lot of people. There are a lot of players/apps that are built off of ExoPlayer. And some big ones like Emby, Plex, Jellyfin, etc. I know Opus has not been a common codec for 5.1 content but it seems to be becoming one of the most popular for its quality and size. Especially when combined with AV1.

Thanks again.

channeladam commented 1 year ago

I just came across this issue on a new Fire TV Stick 4K Max. Am very disappointed that it hasn't been fixed for almost 3 years.

@tonihei , is there any reason why there has been no progress on this? Will there ever be progress? Is it on a roadmap? Or does no one care?

channeladam commented 1 year ago

And also... the YouTube app uses Opus... (you know, YouTube, by GOOGLE)... also has this problem... so anyone who buys an Android platform media player / Google TV and tries to play 5.1 audio through YouTube won't hear the audio as it is meant to be. Is that not incentive enough for Google to fix it?

Simonxdd commented 11 months ago

How is this still not fixed?

johnmolina commented 7 months ago

@tonihei Any update on this? I just bought a brand new Samsung Galaxy Tab S9 with Android 13 and guess what? Multiple apps and even the integrated Samsung video player are all still affected by this bug!

How will there ever be any successful free alternative to DTS and Dolby for multichannel audio if Google can't fix such a fundamental bug.

tonihei commented 7 months ago

We are aware this is still a problem :)

Our hope was that it can be fixed in the actual codec implementation itself, and once it's fixed, we can target a workaround to the Android versions that don't have and need the fix. It seems we won't get a proper fix in the codec anytime soon though.

Another interesting element is that it could be argued that the opus decoders follow the spec too strictly.

The conclusion is that all 3 decoder implementations as a standalone component follow the OPUS spec correctly, but the conversion to Android channel layout still needs to happen.

Why does ffmpeg's opus decoding work then? It only works because ffmpeg defines its own channel layout, which is actually the same as Android's layout. And consequently ffmpeg's opus decoder internally converts the VORBIS layout to its own layout.

Given the connection to VORBIS layouts, I also checked a 5.1 vorbis file and it has the exact same issue with c2.android.vorbis.decoder and OMX.google.vorbis.decoder.

Since this behavior is coupled to the codec implementation and we know it won't be fixed anytime soon (or maybe never given how the actual codec is strictly speaking spec-compliant), I'll see how we can integrate the channel layout mapping into ExoPlayer directly.

channeladam commented 7 months ago

I'll see how we can integrate the channel layout mapping into ExoPlayer directly.

@tonihei thank you, thank you, thank you, thank you, thank you for responding and investigating!

It would be a wonderful end of year gift to all users of Google things (e.g. Android, YouTube app and watchers of any modern media via other players) to have the channel layout mapping in place so we can hear the audio coming from the intended speakers.

Every end-user will agree that it is much more important for the end result to be correct asap than waiting for arguable technical codec fixes that may never happen!

thank you, thank you, thank you!

Do you think the mapping might happen this year?

thank you, thank you, thank you!

tonihei commented 7 months ago

This should be fixed now by the commit above (and will be released as part of the next bugfix and minor release = Media3 1.2.1 and/or 1.3.0)