AudioCapture from Audio Source does not work if source is fed from VideoPlayer

[artoonie]

Describe the bug Trying to capture a scene where the audio is created by a VideoPlayer does not work without some unintuitive workarounds.

Your Setup (please complete the following information): Unity version: 2020.3.13f11 AVPro Movie Capture version: 4.6.5 Operating system version: iOS 14 and Unity Editor on OSX 11.4, Big Sur Capture component used: Any (though, specifically tested CaptureFromScreen and CaptureFromCamera) Capture settings (resolution, frame-rate, codec): Default settings (1080p, 30fps, h264)

To Reproduce In sum: for offline rendering, the VideoPlayer must output audio to the scene, not directly to the speakers. To do this, it must output to an Audio Source. However, AVProMovieCapture cannot hear anything from the Audio Source unless it has a dummy AudioClip set. If no AudioClip is set, the resulting video will have no audio.

Steps:

1. Create a scene with the following configuration (I can provide the demo scene if desired), screenshots below.
   • Camera containing Audio Listener, Audio Source, and Capture Audio from AudioRenderer
   • Video Player with Source set to URL, and choose any video. Render Mode=Camera Near Plane. Track 0 output is set to Audio Source created above.
   • A CaptureFromScreen with AudioSource=Unity and Unity Audio Capture set to Capture Audio created above.
2. With this configuration, play the scene in editor and Start/Stop recording. Notice that no audio is captured.
3. Now, modify Audio Source to have an AudioClip. By disabling "Play on Awake," the clip will never play. Notice that the audio is now captured.

Logs If applicable, add error logs to help explain your problem.

Screenshots GameObject 1: Camera

GameObject 2: Video Player

GameObject 3: Capture Component

I am happy to attach the test scene, though github makes that difficult. The above screenshots should help you quickly reproduce the issue. Note specifically, in screenshot 1, Audio Source > AudioClip = StartRecording is the workaround I've come up with, and it's just an empty audio file.

(By the way, I expect doing this is a bad idea anyway. I just saw a note in AVProVideo that this is not even supported there. I suspect you may advise me here that re-encoding a video in this way is out of scope of this project - if so, I'd welcome such advice. If not, I'd be thrilled to hear how you suggest proceeding.)

[Chris-RH]

Offline rendering is really only recommended for video only. In offline capture mode the video is effectively paused as the frames are stepped through, so no audio is produced. You can however get around it with a bit of fiddling, however the result is not perfect. You would be better off capturing the audio completely separately and syncing it with the video. This is how I recorded audio during offline rendering. Important points: Videoplayer Component: Audio Output: Direct AudioListener Component CaptureAudio (From AudioRenderer) Component: Capture from Screen CaptureFromScreen Component: Audio Source: Unity CaptureFromScreen Component: Frame Rate needs to be changed to get them more in sync

[girish-d]

@Chris-RH

"You would be better off capturing the audio completely separately and syncing it with the video."

What would be the best way to go about this with the latest version of the Movie Capture plugin (I'm on 5.3.3)? I was thinking of doing two passes on the scene - one for audio (possibly using the AudioSourceToWAV component?) and another for the video. Would the merging of the captured audio and video be possible using the plugin?

I'm building for PC/Mac.