Audio support

[jesusmgg]

The ability to include audio from Unity would be a perfect addition. I might work on this and submit a PR if no one else can.

[keijiro]

There were the same requests (#25, #26), but I declined them due to lack of capacity.

Before getting hands dirty for it, I'd recommend checking Unity Recorder. In most cases, Recorder is a superior solution. I'm maintaining this repository for a personal preference.

[jankolkmeier]

Hi there, FFmpegOut also has become my personal preference (thanks @keijiro !), but I needed the audio recording feature as well. So I gave it a quick shot in this commit. Very experimental ;)

I open a TCPListener on the Unity side and have the ffmpeg instance connect to it as a tcp:// input stream. Then I pipe all audio buffers to it for as long as !_terminated. . The port is fixed so one can only use a single Camera Capture at a time (there would be some easy fixes for this though).

There seems to be consistently more audio recorded/written than video, so the resulting video will contain some audio at the end with the video freezing. The difference can be up to a couple of seconds, even on a powerful machine. This is surprising - the number of audio samples should pretty accurately cover the time between enabling and disabling the Camera Capture, so I wonder if the logic in the video pipe actually does a good job at keeping the desired output framerate?

It works good enough for me, for now. But to get this feature to work properly, one should definitely set up a proper test case first (a metronome in the audio track, with some changing visuals driven in sync?)

I tried using "Named Pipes" first, to avoid the tcp connection, but these are very buggy in unity on windows and don't work at all on macOS. I think the easiest and most reliable option is a tcp socket. A completely different approach would be to work with multiple ffmpeg instances and merge them together after the recording is finished.

For best synchronization, one should probably combine the data in Unity into a mts stream with the proper PTS information etc., but I feel that defeats the purpose of FFmpegOut.

[bentaller-lion]

I was going to use your TCP code, but decided it might be way easier to just load an audio file directly?

These parameters load an audio file to the output -i audio.m4a -map 0:v -map 1:a

[jelling]

@jankolkmeier thanks for posting your example code. Just curious, have you done any more audio streaming into ffmpeg? And did you have hands-on issues with named pipes or was that just a general known issue with Unity at the time? I'm seeing things that indicate that may have been fixed but don't know for certain.

[jankolkmeier]

@jelling sorry, haven't touched this in a while. I tried to make the named pipes work on both maxOS and Windows, but wasn't able to get it to work on either - on macOS it seemed like an operating system limitation(?). On windows, I had a test case with the named pipes working in an python script, but couldn't get the same functionality from Unity.

No idea if that got fixed since, sorry.

[Akravator91]

@keijiro Thank you so much for this plugin!!! Unity Recorder is only able to record in editor.., but what i needed is to record the build version! and you have saved mylife with your plugin!!!

@bentaller-lion how did you implement the audio again?

I hope you dont mind to share the code.. :)