[help]Can the receiver get the original voice data of the sender? I want to record and save voice

[shoukailiang]

Currently, both parties are able to communicate by voice. But I would like the receiver to be able to record the sender's voice and play it back later. Is there a corresponding api.

[martindevans]

Dissonance itself does not have a built in way to do this because you can do it using standard Unity features.

If you create an audio filter (https://docs.unity3d.com/ScriptReference/MonoBehaviour.OnAudioFilterRead.html) you get direct access to the audio stream from an AudioSource (read and write, so you can modify the audio too if you like). So you can create a filter to record the voice (into memory or to file, just make sure you don't do any file IO on the audio thread).

Once you have that filter you can create a Dissonance Playback Prefab. Attach to that prefab (in this order):

• VoicePlayback
• AudioSource
• SamplePlaybackComponent
• Your Filter Here

[shoukailiang]

Thank you very much for your reply.I will try later. I see 'IMicrophoneCapture' in the document. Is this capturing local voice？

[martindevans]

IMicrophoneCapture is for replacing the behaviour that gets voice from the microphone and feeds it into Dissonance. e.g. our FMOD Recording package uses this.

If you want to intercept the voice stream that has been recorded and preprocessed by Dissonance refer to these docs: https://placeholder-software.github.io/Dissonance/Tutorials/UsingIMicrophoneSubscriber.html

[shoukailiang]

Thank you very much for your reply.

[shoukailiang]

I'm sorry to bother you again @martindevans . I create a Dissonance Playback Prefab. Audio Renderer is my filter.I drag the prefab into the Playback Prefab field on the Dissonance Comms component inspector. . I have a question

• I can receive voice from remote users, but it seems that the voice is not sent from the audio source? This is my filter code

  using UnityEngine;
  using System;
  using System.IO;
  
  public class AudioRenderer : MonoBehaviour
  {
   // constants for the wave file header
   private const int HEADER_SIZE = 44;
   private const short BITS_PER_SAMPLE = 16;
   private const int SAMPLE_RATE = 44100;
  
   // the number of audio channels in the output file
   private int channels = 2;
  
   // the audio stream instance
   private MemoryStream outputStream;
   private BinaryWriter outputWriter;
  
   // should this object be rendering to the output stream?
   public bool Rendering = false;
  
   public bool stopRecord = false;
  
   private void Update()
   {
       if (stopRecord&&Rendering)
       {
           // save
           Save(@"E:\\1.wav");
           Rendering = false;
       }
   }
  
   /// The status of a render
   public enum Status
   {
       UNKNOWN,
       SUCCESS,
       FAIL,
       ASYNC
   }
  
   /// The result of a render.
   public class Result
   {
       public Status State;
       public string Message;
  
       public Result(Status newState = Status.UNKNOWN, string newMessage = "")
       {
           this.State = newState;
           this.Message = newMessage;
       }
   }
  
   public AudioRenderer()
   {
       this.Clear();
   }
  
   // reset the renderer
   public void Clear()
   {
       this.outputStream = new MemoryStream();
       this.outputWriter = new BinaryWriter(outputStream);
   }
  
   /// Write a chunk of data to the output stream.
   public void Write(float[] audioData)
   {
       // Convert numeric audio data to bytes
       for (int i = 0; i < audioData.Length; i++)
       {
           // write the short to the stream
           this.outputWriter.Write((short)(audioData[i] * (float)Int16.MaxValue));
       }
   }
  
   // write the incoming audio to the output string
   void OnAudioFilterRead(float[] data, int channels)
   {
       if( this.Rendering )
       {
           // store the number of channels we are rendering
           this.channels = channels;
  
           // store the data stream
           this.Write(data);
       }
  
   }
  
   public AudioRenderer.Result Save(string filename)
   {
       Result result = new AudioRenderer.Result();
  
       if (outputStream.Length > 0)
       {
           // add a header to the file so we can send it to the SoundPlayer
           this.AddHeader();
  
           // if a filename was passed in
           if (filename.Length > 0)
           {
               // Save to a file. Print a warning if overwriting a file.
               if (File.Exists(filename))
                   Debug.LogWarning("Overwriting " + filename + "...");
  
               // reset the stream pointer to the beginning of the stream
               outputStream.Position = 0;
  
               // write the stream to a file
               FileStream fs = File.OpenWrite(filename);
  
               this.outputStream.WriteTo(fs);
  
               fs.Close();
  
               // for debugging only
               Debug.Log("Finished saving to " + filename + ".");
           }
  
           result.State = Status.SUCCESS;
       }
       else
       {
           Debug.LogWarning("There is no audio data to save!");
  
           result.State = Status.FAIL;
           result.Message = "There is no audio data to save!";
       }
  
       return result;
   }
  
   /// This generates a simple header for a canonical wave file, 
   /// which is the simplest practical audio file format. It
   /// writes the header and the audio file to a new stream, then
   /// moves the reference to that stream.
   /// 
   /// See this page for details on canonical wave files: 
   /// http://www.lightlink.com/tjweber/StripWav/Canon.html
   private void AddHeader()
   {
       // reset the output stream
       outputStream.Position = 0;
  
       // calculate the number of samples in the data chunk
       long numberOfSamples = outputStream.Length / (BITS_PER_SAMPLE / 8);
  
       // create a new MemoryStream that will have both the audio data AND the header
       MemoryStream newOutputStream = new MemoryStream();
       BinaryWriter writer = new BinaryWriter(newOutputStream);
  
       writer.Write(0x46464952); // "RIFF" in ASCII
  
       // write the number of bytes in the entire file
       writer.Write((int)(HEADER_SIZE + (numberOfSamples * BITS_PER_SAMPLE * channels / 8)) - 8);
  
       writer.Write(0x45564157); // "WAVE" in ASCII
       writer.Write(0x20746d66); // "fmt " in ASCII
       writer.Write(16);
  
       // write the format tag. 1 = PCM
       writer.Write((short)1);
  
       // write the number of channels.
       writer.Write((short)channels);
  
       // write the sample rate. 44100 in this case. The number of audio samples per second
       writer.Write(SAMPLE_RATE);
  
       writer.Write(SAMPLE_RATE * channels * (BITS_PER_SAMPLE / 8));
       writer.Write((short)(channels * (BITS_PER_SAMPLE / 8)));
  
       // 16 bits per sample
       writer.Write(BITS_PER_SAMPLE);
  
       // "data" in ASCII. Start the data chunk.
       writer.Write(0x61746164);
  
       // write the number of bytes in the data portion
       writer.Write((int)(numberOfSamples * BITS_PER_SAMPLE * channels / 8));
  
       // copy over the actual audio data
       this.outputStream.WriteTo(newOutputStream);
  
       // move the reference to the new stream
       this.outputStream = newOutputStream;
   }
  }

[martindevans]

I can receive voice from remote users

I'm not completely sure I understand what you're asking, sorry. Do you mean you can hear voice, but your filter is not receiving it?

it seems that the voice is not sent from the audio source?

That's the correct. The AudioSource is playing back silence and then the SamplePlaybackComponent is an audio filter which injects the voice audio. This is a bit weird, but it's the lowest latency way to do it.

A far as I'm aware this shouldn't be a problem for you though? Audio flows down the gameobject, so as long as your filter is below the SamplePlaybackComponent you should be receiving the voice data in your filter.

This is my filter code

This looks broadly ok, but there are a few details that might not be right:

private const int SAMPLE_RATE = 44100; The sample rate of the audio is determined by whatever sample rate the Unity audio system wants to run at. You can read this from the AudioSettings object (https://docs.unity3d.com/ScriptReference/AudioSettings.html)

void OnAudioFilterRead This method is called on the audio thread, not the main thread. I don't think the way you're writing to the outputWriter on the audio thread and reading from it on the main thread is safe.

[shoukailiang]

Thank you very much. It's OK now.

This method is called on the audio thread, not the main thread. I don't think the way you're writing to the outputWriter on the audio thread and reading from it on the main thread is safe.

which mean the following code is problematic.

// main thread 
     private void Update()
     {  
         // Record 20 seconds for example
        time += Time.deltaTime;
        if (Rendering&&time >=20)
        {
            // save
            Save(@"E:\\1.wav");
            Debug.Log("save success");
            Rendering = false;
        }
     }

// audio thread 
     void OnAudioFilterRead(float[] data, int channels)
     {      
         if( this.Rendering)
         {
             // store the number of channels we are rendering
             this.channels = channels;

             // store the data stream
             this.Write(data);   
         }
     }

Therefore, do I need to put the all IO operation into the main thread？

[martindevans]

IO operation into the main thread？

You definitely don't want to do any IO on the audio thread - it can't be stalled under any circumstances!

If I'm understanding your current code correctly you're writing to an MemoryStream inside OnAudioFilterRead (audio thread) and then in Update (main thread) you're saving that to disk. That's not too bad (it might stall the main thread, which isn't great, but it's much better than stalling the audio thread).

However, the one problem is the writing and the reading are not threadsafe, so you may end up with corrupted state. You either need a threadsafe collection to transfer the data, or a lock to protect the datastructures. If you go with a lock you must hold it for the absolutely shortest time possible (e.g. grab the stream, put a new one in place, release the lock).

[shoukailiang]

Thanks!!!!