search

RenderHeads / UnityPlugin-AVProVideo

AVPro Video is a multi-platform Unity plugin for advanced video playback
https://www.renderheads.com/products/avpro-video/
232 stars 28 forks source link

Multi-channel audio not received by AudioOutput component on Windows #158

Open OlafZwe opened 5 years ago

OlafZwe commented 5 years ago

Describe the bug On Windows 10, using Media Foundation with Audio Mode Unity and deselecting Stereo in Unity Audio settings. When playing a video with a multi-channel audio track (e.g. 4 channels), the AudioOutput component attached to the media player only receives silence in its OnAudioFilterRead function. The audio is still audible, but appears to be playing in System Direct mode rather than through Unity. See screenshot below for setup.

We found this bug while developing a custom AudioOutput component that should take the raw multi-channel audio data and pass it through to the Audio360 SDK's NativeSpatDecoderQueue in order to be spatialized. To be able to handle (adaptive streaming) spatial audio, we need to be able to access the raw multi-channel audio data directly, regardless of Windows' speaker setup or Unity's speaker/audio setup. Is this possible?

This bug might be related to issue #72 , but that issue did appear to describe a different situation.

Your Setup (please complete the following information):

To Reproduce

  1. Use the GameObject > AVPro Video > Media Player with Unity Audio menu option
  2. Change the Media Player (and AudioOutput) settings according to the screenshot below
  3. Play a (local) video with multi-channel audio (e.g. 4 channels)
  4. Notice that changing the AudioSource settings or disabling the component altogether does not change or stop the audio from playing.

Screenshots image

Videos Example video with 4-channel spatial audio

owlboy commented 4 years ago

Hello! I have been struggling with this issue since 2018. As that post states, this appears to be something that was supported in the past. Is this incorrect?

Is there any information that can be shared about this topic?

Can you elaborate on what you said in this comment on #72?

This component might not help in all cases, eg being able to direct audio channels to specific Transforms, but this could act as a reference to make a new component to achieve this.

My Goal:

image

Sample Videos:

My Setup:

thank you!

AndrewRH commented 4 years ago

Thanks both of you for sharing such detailed information. We'll look into this with your files and use cases in mind and report back soon.

fingerx commented 4 years ago

@AndrewRH any news?thanks

AndrewRH commented 4 years ago

@OlafZwe Hi, I know this is rather late, but thanks for the excellent example. We have replicated the issue and are working on a fix.

@owlboy I believe this is a slightly different issue to the original poster, but we're looking into it.

fingerx commented 4 years ago

Hope it can be fixed soon. thank.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 4 years ago

This issue has been automatically closed because it has not had recent activity. If you wish to continue this issue, then please create a new issue as we do not monitor closed issues.

fingerx commented 3 years ago

@AndrewRH any news for that?

cross6180 commented 3 years ago

@AndrewRH I am trying to do this with the Android platform. I get all the 5.1 channel raw data from the GrabAudio() function in the AudioOutputManager.cs, but find out that the data array is arranged differently from Windows (look like it is orderless). Is there something I missed? Thank you.

My Setup: Unity version: 2019.4.10f1 AVPro Video version: v2.1.6 trial version Operating system version: Android 10 Device model: VR HMD Video specs (resolution, frame-rate, codec, file size): mp4 video container with H.264 codec and a single 6-channel audio track

AndrewRH commented 2 years ago

Going to look into this soon for Windows, currently I think it works okay if you're using multi-channel PCM audio. For AAC compressed audio there are internal limits in the Windows decoder for number of channels. I will be investigating further soon.

Ste-RH commented 2 years ago

@AndrewRH I am trying to do this with the Android platform. I get all the 5.1 channel raw data from the GrabAudio() function in the AudioOutputManager.cs, but find out that the data array is arranged differently from Windows (look like it is orderless). Is there something I missed? Thank you.

My Setup: Unity version: 2019.4.10f1 AVPro Video version: v2.1.6 trial version Operating system version: Android 10 Device model: VR HMD Video specs (resolution, frame-rate, codec, file size): mp4 video container with H.264 codec and a single 6-channel audio track

On Android…I believe this was a bug that we fixed a month or so ago. Do grab the latest version and see if the issue is addressed.

gabrielstuff commented 2 years ago

Hello, Any news on this issue ? It would be great to know the status ?

Thanks a lot !

andriworld commented 1 year ago

For anyone trying to play each audio channel of a 5.1 audio track in a separate Unity AudioSource (as in https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/158#issuecomment-594321016) here's how I configured each speaker object: image

Tested on Windows and Android, Unity 2021.3.9f1, AVPro v2.6.7.
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 92 kb/s (default)