FMOD support

[joonjoonjoon]

I raised this in the wrong repo earlier, but...

Fmod is a popular and high-end audio engine plugin for Unity. Unity uses a forked (dated) version of FMOD under the hood but it is not compatible with FMOD studio which sound designers like to use. FMOD as a plugin entirely replaces Unity's audio system and is incompatible with it.

The Unity video capture plugin now allows recording audio in offline mode, which is perfect, but is not compatible with FMOD.

https://www.fmod.com/unity

I initially posted here, but got a bit confused about stuff. I hope this issue makes more sense and is in the right place. https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/526#issuecomment-739893476

[AndrewRH]

Thanks, I see - I guess this is a bit like WWISE - they also have a plugin that can be used for all audio in Unity and it's relatively popular.. We did look at writing an audio capture solution for WWISE previously but ran into some issues.

I'm not familiar with the FMOD plugin for Unity so this is something we would have to investigate.

[ChadKeating]

@AndrewRH is offline audio now supported for wwise/fmod with this update?: https://www.renderheads.com/content/docs/AVProMovieCapture/versions/4.6.0.html

[chadefranklin]

Yes, FMOD support would be very helpful!

[robinW-D]

Hiya, any update on FMOD support?

[Chris-RH]

nothing as of yet

[Grhyll]

Hi, just for anyone who could use that info, based on:

• This example from FMOD: https://fmod.com/docs/2.02/unity/examples-dsp-capture.html
• And this post in issues: https://github.com/RenderHeads/UnityPlugin-AVProMovieCapture/issues/282 I managed to have fmod roughly work with AVProMovieCapture (only tested in editor so far). You can even chose to record only a specified channel group instead of the master one.

There are still a few things I don't really understand, and some optimization needed, but as a starting point:

• Set the Audio Source to Manual in the CaptureFromScreen (or other capture script).
• Set the following: capture.ManualAudioChannelCount = 2; // It doesn't seem to work with any other value capture.ManualAudioSampleRate = 48000; // It probably depends on settings in Fmod, mine is set to 48K
• Then add this script, and call its relevant methods:

  using System;
  using System.Collections.Generic;
  using UnityEngine;
  using System.Runtime.InteropServices;
  using RenderHeads.Media.AVProMovieCapture;

  public class CaptureAudioFromFmodDsp : MonoBehaviour
  {
      //public int SampleRate => mSampleRate;
      //public int ChannelCount => mChannels;

      const string busToRecord = "bus:/Ingame/SFX";
      CaptureFromScreen capture;

      private FMOD.DSP_READ_CALLBACK mReadCallback;
      private FMOD.DSP mCaptureDSP;
      private float[] mDataBuffer;
      private GCHandle mObjHandle;
      private uint mBufferLength;

      [AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READ_CALLBACK))]
      static FMOD.RESULT CaptureDSPReadCallback(ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels)
      {
          FMOD.DSP_STATE_FUNCTIONS functions = (FMOD.DSP_STATE_FUNCTIONS)Marshal.PtrToStructure(dsp_state.functions, typeof(FMOD.DSP_STATE_FUNCTIONS));

          IntPtr userData;
          functions.getuserdata(ref dsp_state, out userData);
          GCHandle objHandle = GCHandle.FromIntPtr(userData);

          //int sampleRate = 0;
          //functions.getsamplerate(ref dsp_state, ref sampleRate);

          CaptureAudioFromFmodDsp obj = objHandle.Target as CaptureAudioFromFmodDsp;

          // Copy the incoming buffer to process later
          int lengthElements = (int)length * inchannels;
          Marshal.Copy(inbuffer, obj.mDataBuffer, 0, lengthElements);
          if (obj.capture != null && obj.capture.IsCapturing() && !obj.capture.IsPaused())
          {
              float[] t = new float[(int)length * 2];
              for (int i = 0; i < length; i++)
              {
                  if (inchannels >= 2)
                  {
                      for (int j = 0; j < 2; j++)
                      {
                          t[i * 2 + j] = obj.mDataBuffer[i * inchannels + j];
                      }
                  }
                  else
                  {
                      t[i * 2] = obj.mDataBuffer[i];
                      t[i * 2 + 1] = 0;
                  }
              }
              obj.capture.EncodeAudio(t);
          }

          // Copy the inbuffer to the outbuffer so we can still hear it
          Marshal.Copy(obj.mDataBuffer, 0, outbuffer, lengthElements);

          return FMOD.RESULT.OK;
      }

      void RegisterToReadCallback()
      {
          // Assign the callback to a member variable to avoid garbage collection
          mReadCallback = CaptureDSPReadCallback;

          // Allocate a data buffer large enough for 8 channels, pin the memory to avoid garbage collection
          uint bufferLength;
          int numBuffers;
          FMODUnity.RuntimeManager.CoreSystem.getDSPBufferSize(out bufferLength, out numBuffers);
          mDataBuffer = new float[bufferLength * 8];
          mBufferLength = bufferLength;

          // Get a handle to this object to pass into the callback
          mObjHandle = GCHandle.Alloc(this);
          if (mObjHandle != null)
          {
              // Define a basic DSP that receives a callback each mix to capture audio
              FMOD.DSP_DESCRIPTION desc = new FMOD.DSP_DESCRIPTION();
              desc.numinputbuffers = 1;
              desc.numoutputbuffers = 1;
              desc.read = mReadCallback;
              desc.userdata = GCHandle.ToIntPtr(mObjHandle);

              // Create an instance of the capture DSP and attach it to the wanted channel group to capture its audio
              //FMOD.ChannelGroup masterCG;
              //if (FMODUnity.RuntimeManager.CoreSystem.getMasterChannelGroup(out masterCG) == FMOD.RESULT.OK)

              FMOD.Studio.Bus bus = FMODUnity.RuntimeManager.GetBus(busToRecord);
              bus.lockChannelGroup(); // Force to create the channel group
              FMODUnity.RuntimeManager.StudioSystem.flushCommands(); // Force the channel group creation to happen right now
              if (bus.getChannelGroup(out FMOD.ChannelGroup cg) == FMOD.RESULT.OK)
              {
                  if (FMODUnity.RuntimeManager.CoreSystem.createDSP(ref desc, out mCaptureDSP) == FMOD.RESULT.OK)
                  {
                      if (cg.addDSP(0, mCaptureDSP) != FMOD.RESULT.OK)
                      {
                          Debug.LogWarningFormat("FMOD: Unable to add mCaptureDSP to the master channel group");
                      }
                  }
                  else
                  {
                      Debug.LogWarningFormat("FMOD: Unable to create a DSP: mCaptureDSP");
                  }
              }
              else
              {
                  Debug.LogWarningFormat("FMOD: Unable to create a channel group from sfxBus, isValid: " + bus.isValid() + ", return " + bus.getChannelGroup(out cg));
              }
              bus.unlockChannelGroup();
          }
          else
          {
              Debug.LogWarningFormat("FMOD: Unable to create a GCHandle: mObjHandle");
          }
      }

      void OnDestroy()
      {
          UnregisterFromReadCallback();
      }
      void UnregisterFromReadCallback()
      {
          if (mObjHandle.IsAllocated)
          {
              FMOD.Studio.Bus bus = FMODUnity.RuntimeManager.GetBus(busToRecord);
              bus.lockChannelGroup();
              FMODUnity.RuntimeManager.StudioSystem.flushCommands();
              if (bus.getChannelGroup(out FMOD.ChannelGroup cg) == FMOD.RESULT.OK)
              {
                  if (mCaptureDSP.hasHandle())
                  {
                      cg.removeDSP(mCaptureDSP);

                      // Release the DSP and free the object handle
                      mCaptureDSP.release();
                  }
              }
              bus.unlockChannelGroup();
              mObjHandle.Free();
          }
      }

      public void PrepareCapture()
      {
      }
      public void StartCapture(CaptureFromScreen capture)
      {
          RegisterToReadCallback();
          this.capture = capture;
      }
      public void StopCapture()
      {
          UnregisterFromReadCallback();
          capture = null;
      }
      public void PauseCapture()
      {
      }
      public void ResumeCapture()
      {
      }
  }

(Sorry the "code" markdown doesn't seem to work correctly!)

[Chris-RH]

Thanks you. Interested to see if it works in build :)

[Grhyll]

Looks like it works fine in build as well :)

[Chris-RH]

Awesome, that's great, thanks for letting us know 😀

On Wed, 12 Jul 2023, 11:16 Grhyll, @.***> wrote:

Looks like it works fine in build as well :)

— Reply to this email directly, view it on GitHub https://github.com/RenderHeads/UnityPlugin-AVProMovieCapture/issues/81#issuecomment-1632234774, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYRROUIU4P5O4ME3M22NTYLXPZ2OPANCNFSM4UTXGRDQ . You are receiving this because you commented.Message ID: @.*** com>

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.