[help] how to record the audio and playback to everyone in room

[Asaadmk]

here is the workflow of what i want

the user press a button one time and start speaking, the voice will go to everyone in the room (this is already i am doing successfully without the button press part) , then he/she press the button again and what the user said should be recorded to an audio clip

then i want this audio clip that was just recorded to be submitted to everyone in the room again

thanks in advance

[martindevans]

Hi Asaadmk,

The first part should be fairly simple. You can enable/disable the broadcast trigger when the button is pressed, or you can directly open and close a channel to the room.

The second part is a little more complex. I think for this you're probably going need to use a pretty new feature we added recently, custom microphone plugins. This allows you to replace the microphone which Dissonance gets it's audio from with your own piece of code which gets the audio from wherever you want. You'll want to have some kind of control which decides if it provides audio from the microphone or if it is providing audio from an audio file. For reference look into BasicMicrophoneCapture, this is the default implementation of the microphone capture interface. Also check out WaveFileWriter which writes out wav files to disk (we don't have a reader, have a look at NAudio to see if you can use theirs).

When you're playing back the file you will want to make sure that you're actually broadcasting the voice, to do that simply open a channel to the room for the duration of the file playback.

[Asaadmk]

thanks that does put me on the right track;

but i still will have to do a lot of testing since i don't really know how it works,

can i add something like this in the same basic microphone capture ?

// called when the user click the button, or would this not work because the mic is already being recorded into  the _clip?

public void CreateRecordedClip()
{
  RecordedClip = Microphone.Start(_micName, false, 10, 44100);
}

// called when i want to playback the recorded clip to every one in room
public void DrainRecordedClip()
{
  _clip = RecordedClip;
  DrainMicSamples();
}

sorry i know i am asking for too much help , but this would take me a lot of time to figure out while it will be mush easier for you

[martindevans]

can i add something like this in the same basic microphone capture ?

I would avoid actually modifying the BasicMicrophoneCapture itself - that'll make upgrading to future versions of Dissonance difficult if any of the implementation details changes. Instead I would implement a new wrapper class which implements IMicrophoneCapture (docs here and wraps basic microphone capture. When you're recording from the microphone you can just pass the calls through to the real BasicMicrophoneCapture. When you want to playback some audio of your own you should then:

1. On the next call to Update return true (this forces a "reset" of the capture system, which will prepare it for your new audio).
2. StopCapture will be called as part of the reset, call StopCapture on your wrapped BasicMicrophoneCapture to stop it recording audio.
3. A short time later StartCapture will be called, you can ignore the mic name. Return the format of your recorded data. Loop through subscribers and call Reset on them.
4. Every time update is called pull out some audio from your recording, loop through all the subscribers and pass the data to them with ReceiveMicrophoneData
5. Once your file playback is done return true to force a reset again. Now switch back to true microphone capture (similar to steps 1-3) again.

Hopefully that makes sense :)

[Asaadmk]

yes, thanks kind off

but what do you mean by

"On the next call to Update return true" , which class your taking about in this statement

[martindevans]

In everything above I am referring to your implementation of IMicrophoneCapture. The update method is called by Dissonance every frame and is where you pass audio data on to subscribers - if it ever returns true Dissonance immediately stops the microphone and will restart it (after a short delay). This stop/start cycle resets the entire audio capture pipeline (e.g. preparing it for a new audio source). Here's the section on the update method quoted from the docs:

bool Update()

• Pass buffered data on to the subscribers.
• Returns true if the microphone needs to be reset. In this case Stop will immediately be called (and start may be called afterwards).

[Asaadmk]

OK, thanks you are as awesome as your asset

but again i am stuck , i figured that i can't call microphone.start to record for 15 sec when the user click the button

since the user will be speaking in the global chat with voice activation the microphone is always working but i only need to record what the user is saying only when he clicks the button

so i created the wrapper microphone capture class and added a function to be called when the user click the button to start adding the audio samples into a float array

public class MyMicrophpneCapture :   MonoBehaviour, IMicrophoneCapture
{
    int minFreq;
    int maxFreq;
    int sampleRate;
    float RecordingPeriod=20;
    string micName;

    bool Record = false;
    AudioClip _clip;
    BasicMicrophoneCapture BasicCapture = new BasicMicrophoneCapture() ;
    public AudioSource audsource;
    public bool IsRecording
    {

        get
        {
            return BasicCapture.IsRecording;
        }
    }

    public TimeSpan Latency
    {
        get
        {
            return BasicCapture.Latency;
        }
    }

    public WaveFormat StartCapture(string name)
    {
        micName = name;
        Microphone.GetDeviceCaps(name, out minFreq, out maxFreq);
         sampleRate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);
        return    BasicCapture.StartCapture (name);
    }

    public void StopCapture()
    {
         BasicCapture.StopCapture();
    }

    public void Subscribe(IMicrophoneSubscriber listener)
    {
        BasicCapture.Subscribe(listener);
    }

    public bool Unsubscribe(IMicrophoneSubscriber listener)
    {
 return       BasicCapture.Unsubscribe(listener);
    }

    public bool UpdateSubscribers()
    {

        return   BasicCapture.UpdateSubscribers();
    }

   public void StartRecording()
    {
        //_clip = Microphone.Start(micName, false, RecordingPeriod, sampleRate);
        //EndRecording(BasicCapture._clip, micName);
        BasicCapture.RecordedClipData = new float[0];
        BasicCapture.StartRecording = true;
        Timing.RunCoroutine(StopRecordingAfterSecs(), MEC.Segment.LateUpdate);
    }

    IEnumerator<float> StopRecordingAfterSecs()
    {

            yield return Timing.WaitForSeconds(RecordingPeriod);

        BasicCapture.StartRecording = false;
        _clip = AudioClip.Create("RecordedClip", BasicCapture.RecordedClipData .Length, BasicCapture._clip.channels, BasicCapture._clip.frequency, false);
        _clip.SetData(BasicCapture.RecordedClipData, 0);        //Give it the data from the old clip
        audsource.clip = _clip;
        audsource.Play();
    }

for now i don't really know how to get the audio clip samples without modifying the basic capture but i can't even get it to work in the first place ,

i have modified the drain mic sample to see if it is going to work but i am still trying

 //Inform the buffer how many samples we want to read
            //Debug.Log(samplesToRead);
            _readBuffer.Alloc(samplesToRead);
            try
            {
                while (samplesToRead > 0)
                {
                    //Read from mic
                    var buffer = _readBuffer.GetBuffer(ref samplesToRead, true);
                    _clip.GetData(buffer, _readHead);
                    //my microphone capture addtion 
                    if (StartRecording)
                    {
                        LastPostion = RecordedClipData.Length;
                        if (LastPostion != 0)
                            LastPostion += 1;
                        Debug.Log("LastPostion" + LastPostion);
                        //Debug.Log("buffer.Length" + buffer.Length);
                        Array.Resize(ref RecordedClipData, RecordedClipData.Length + buffer.Length);
                        Debug.Log(" RecordedClipData.Length" + RecordedClipData.Length);
                        for (int i = 0; i < buffer.Length; i++)
                        {
                            if (LastPostion + i > RecordedClipData.Length)
                            {
                                Debug.Log("LastPostion" + LastPostion);
                                Debug.Log("i : " + i);
                                Debug.Log("LastPostion  + i  " + LastPostion + i);
                                break;
                            }
                            Debug.Log("LastPostion  + i  " + LastPostion + i);
                            RecordedClipData[LastPostion  + i] = buffer[i];

                        }

                    } 

is this the way to go ?

[martindevans]

Sorry I got wrapped in discussing how to play back your recordings and never mentioned how to actually record! Recording should actually be fairly simple and require no modifications at all.

When the record button is clicked you can do something like this:

// Start recording
DissonanceComms comms;
comms.MicrophoneCapture.Subscribe(myRecorderObject);

// Stop recording
comms.MicrophoneCapture.Unsubscribe(myRecorderObject);

This will supply the raw microphone samples to your recorder object. This is actually what Dissonance itself does - the microphone is always running but the encoder subscribes/unsubscribes itself when audio needs to be transmitted.

Your recorder object just needs to implement the Assets/Plugins/Dissonance/Core/Audio/Capture/IMicrophoneSubscriber interface which is a pretty simple interface.

[Asaadmk]

thanks the voice recording is working after your latest response, now i am testing transmitting the voice but something weird is happening , i am ruining 2 clients from my laptop and in normal mode it is working fine, but when i start sending the voice i have to keep switching between clients , or the voice capture will not start again ;

and sometime the recording sending works fine but mostly it does not or part of it plays on the other subscribers ;

and i get these warnings

Detected a frame skip, forcing capture pipeline reset insufficient buffer space, requested 98880, clamped to 16383 Lost {0} samples in the preprocessor (buffer full), injecting silence to compensate

here is my microphone capture class

using Dissonance.Audio.Capture;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using NAudio.Wave;
using Dissonance.Datastructures;
using System;
using Dissonance;
using System.Linq;
using MEC;

public class MyMicrophpneCapture :   MonoBehaviour, IMicrophoneCapture
{

    float RecordingPeriod=15;
    int ReadIndex=0;
    int LastBufferSize = 0;
    int ReadBufferSize=1000;
    bool ResetCapture=false;
    bool PlayRecording = false;
    ArraySegment<float> ReadBuffer;
    bool Record = false;
    AudioClip _clip;
    BasicMicrophoneCapture BasicCapture = new BasicMicrophoneCapture() ;
    MyMicrophoneSubscriber MicrophoneSubscriber = new MyMicrophoneSubscriber();
    private readonly List<IMicrophoneSubscriber> _subscribers = new List<IMicrophoneSubscriber>();
    DissonanceComms comms;
    public AudioSource audsource;

    private void Start()
    {
        comms = gameObject.GetComponent<DissonanceComms>();
    } 
    public bool IsRecording
    {

        get
        {
            return BasicCapture.IsRecording;
        }
    }

    public TimeSpan Latency
    {
        get
        {
            return BasicCapture.Latency;
        }
    }

    public WaveFormat StartCapture(string name)
    {
        Debug.Log("StartCapture");
        if (ResetCapture )
        {
            for (var i = 0; i < _subscribers.Count; i++)
                _subscribers[i].Reset ();
            ResetCapture = false;
            return MicrophoneSubscriber.SamplesFormat;
                }

        return    BasicCapture.StartCapture (name);
    }

    public void StopCapture()
    {
        Debug.Log("StopCapture");
        BasicCapture.StopCapture();
    }

    public void Subscribe(IMicrophoneSubscriber listener)
    {
        if (listener == null) throw new ArgumentNullException("listener");

        _subscribers.Add(listener);
        BasicCapture.Subscribe(listener);
    }

    public bool Unsubscribe(IMicrophoneSubscriber listener)
    {
        if (listener == null) throw new ArgumentNullException("listener");

         _subscribers.Remove(listener);
        return       BasicCapture.Unsubscribe(listener);
    }

    public bool UpdateSubscribers()
    {
        if (ResetCapture)
            return true;

        if (PlayRecording)
        {
            Debug.Log("PlayRecording");
            Debug.Log("MicrophoneSubscriber .Localbuffer .Count  -1"  + (MicrophoneSubscriber.Localbuffer.Count - 1).ToString ());
            //while (ReadIndex <= MicrophoneSubscriber .Localbuffer .Count  -1 )
            //{
                //Read from mic
                //ReadLastIndex = ReadIndex ;

                if (ReadIndex + ReadBufferSize > MicrophoneSubscriber.Localbuffer.Count - 1)
                {
                    LastBufferSize = (MicrophoneSubscriber.Localbuffer.Count - 1) - ReadIndex;
                    SendSegment(new ArraySegment<float>(MicrophoneSubscriber.Localbuffer.GetRange(ReadIndex, LastBufferSize).ToArray()),MicrophoneSubscriber .SamplesFormat );
                }
                else
                {

                    SendSegment(new ArraySegment<float>(MicrophoneSubscriber.Localbuffer.GetRange(ReadIndex, ReadBufferSize).ToArray()), MicrophoneSubscriber.SamplesFormat);

                //}

            }
            ReadIndex += ReadBufferSize + 1;

            if (ReadIndex >= MicrophoneSubscriber.Localbuffer.Count - 1)
            {
                PlayRecording = false;
                return true;
            }
            return false;
        }
        else

        return   BasicCapture.UpdateSubscribers();
    }

    void SendSegment(ArraySegment<float> ReadBuffer , WaveFormat Format)
    {
        for (var i = 0; i < _subscribers.Count; i++)
            _subscribers[i].ReceiveMicrophoneData(ReadBuffer, Format);
    }
   public void StartRecording()
    {
        Debug.Log("recording started");
        MicrophoneSubscriber.Reset();
        comms.MicrophoneCapture.Subscribe(MicrophoneSubscriber);
        Timing.RunCoroutine(StopRecordingAfterSecs(), MEC.Segment.LateUpdate);
    }

    public void SendRecording()
    {
        ResetCapture = true;
        PlayRecording = true;
    }

    IEnumerator<float> StopRecordingAfterSecs()
    {

            yield return Timing.WaitForSeconds(RecordingPeriod);
        Debug.Log("recording endded");
        Debug.Log(MicrophoneSubscriber.getSamples() .Length);
        Debug.Log(MicrophoneSubscriber.SamplesArray.Length);

      comms.MicrophoneCapture.Unsubscribe(MicrophoneSubscriber);
        //_clip = AudioClip.Create("RecordedClip", MicrophoneSubscriber.SamplesArray.Length, MicrophoneSubscriber.SamplesFormat .Channels, MicrophoneSubscriber.SamplesFormat .SampleRate, false);
        //_clip.SetData(MicrophoneSubscriber.SamplesArray, 0);        //Give it the data from the old clip
        //audsource.clip = _clip;
        //Debug.Log("_clip.samples " + _clip.samples);
        //audsource.Play();
    }

}

And microphone subscriber class

using Dissonance.Audio.Capture;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using NAudio.Wave;
using System;

public class MyMicrophoneSubscriber : MonoBehaviour, IMicrophoneSubscriber
{
  public  List<float> Localbuffer=new List<float> ();
    public float[] SamplesArray;
    public WaveFormat SamplesFormat;
    public void ReceiveMicrophoneData(ArraySegment<float> buffer, WaveFormat format)
    {

        Localbuffer .AddRange(buffer.Array) ;
        SamplesFormat = format;

    }

    public void Reset()
    {
        SamplesArray = new float[0];
        Localbuffer = new List<float>();
    }

   public float[] getSamples()
    {
        return SamplesArray = Localbuffer.ToArray();
    }
}

[martindevans]

You microphone recorder class has a couple of issues:

• The buffer supplied to you in ReceiveMicrophoneData is a segment of a buffer, you need to only add the part of the array specified by the offset and the count properties.
• In reset you should probably just clear the list instead of creating a new one (reduce garbage collections)

Your capture class also has some issues:

• Your implementation of IsRecording only returns if the wrapped BasicMicrophoneCapture is recording. But when you're playing back your recorded audio you want to return true for this.
• StartRecording and StopRecording look very complex, but all they need to do is subscribe and unsubscrube your microphone subscriber to the wrapped mic capture.
• I'm not quite sure what your playback is doing, all you need to do is look at the sample rate and how much time has passed since the last frame of playback and return the appropriate number of samples.

e.g. if your recording has a sample rate of 1000Hz and 0.1s has past you just pass 0.1 * 1000 = 100 samples to all the subscribers. You can take advantage of the array segment to save yourself allocating a load. It's perfectly safe to do:

int sent_samples;
float[] recorded_data;

//Each frame...
var samples = time_since_last_frame * recording_sample_rate;
SendToSubscribers(new ArraySegment(recorded_data, sent_samples, samples));
sent_samples += samples;

You need to perform a switchover from returning mic data to returning recorded data. That will look a bit like this:

1. Decide to start playing a recording
2. Next time Update() is called on the microphone, return true to force a reset
3. StopCapture will be called
4. StartCapture will be called, return the format of the recorded data
5. Play back the data (see above)
6. Once playback is complete, return true from the next update to force a reset
7. StopCapture will be called
8. StartCapture will be called, call it on the BasicMicrophoneCapture instance
9. Now just pass all calls through to the BasicMicrophoneCapture until the next time you want to play back a recording

In general you shouldn't need to be calling anything on the wrapped BasicMicrophoneCapture except passing through the calls defined in the IMicrophoneCapture interface.

[Asaadmk]

thank you very much it is working as expected, you pretty much wrote the whole thing for me,

below is the code if anyone have a use for it, i added some comments and did some cleaning , i might still be doing something stupid/ not optimized if you spotted something please let me know:

using Dissonance.Audio.Capture;
using System.Collections.Generic;
using UnityEngine;
using NAudio.Wave;
using System;
using Dissonance;
using MEC;

public class MyMicrophpneCapture :   MonoBehaviour, IMicrophoneCapture
{
    int sent_samples;
    int recording_sample_rate=0;
    float RecordingPeriod=15;
    bool ResetCapture=false;
    bool PlayRecording = false;

    BasicMicrophoneCapture BasicCapture = new BasicMicrophoneCapture() ;
    MyMicrophoneSubscriber MicrophoneSubscriber = new MyMicrophoneSubscriber();
    private readonly List<IMicrophoneSubscriber> _subscribers = new List<IMicrophoneSubscriber>();
    DissonanceComms comms;

    private void Start()
    {
        comms = gameObject.GetComponent<DissonanceComms>();
    } 
    public bool IsRecording
    {

        get
        {
            // if playing the recording to the players in the room return true otherwise route to the normal BasicCapture 
            if (PlayRecording)
                return true;
            else
                return BasicCapture.IsRecording;
        }
    }

    public TimeSpan Latency
    {
        get
        {
            return BasicCapture.Latency;
        }
    }

    public WaveFormat StartCapture(string name)
    {   Debug.Log("startCapture");

        /////////   calcluting the sample rate to be used in update subscribers 
        int minFreq;
        int maxFreq;
        Microphone.GetDeviceCaps(name, out minFreq, out maxFreq);
        recording_sample_rate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);
        //////////  calcluting the sample rate to be used in update subscribers 

        // reset subscribers is already being done though stop recording therefor i am only returning the recording format
        if (ResetCapture )
        {
            ResetCapture = false;
            return MicrophoneSubscriber.SamplesFormat;
                }

        return    BasicCapture.StartCapture (name);
    }

    public void StopCapture()
    {
        Debug.Log("stopCapture");

        BasicCapture.StopCapture();
    }

    public void Subscribe(IMicrophoneSubscriber listener)
    {
        if (listener == null) throw new ArgumentNullException("listener");

        _subscribers.Add(listener);
        BasicCapture.Subscribe(listener);
    }

    public bool Unsubscribe(IMicrophoneSubscriber listener)
    {
        if (listener == null) throw new ArgumentNullException("listener");

         _subscribers.Remove(listener);
        return       BasicCapture.Unsubscribe(listener);
    }

    public bool UpdateSubscribers()
    {

        if (ResetCapture)
            return true;

        if (PlayRecording)
        {

           // detarimain the sample size to be sent to players
            var samples = Time.deltaTime * recording_sample_rate;

            // check if the last sample is bigger than the array length and minimize it to the actual size
            if (samples + sent_samples > MicrophoneSubscriber.SamplesArray.Length)
                samples = MicrophoneSubscriber.SamplesArray.Length - sent_samples;

        // send the sample to the subscribers
            SendSegmentToSubscribers(new ArraySegment<float>(MicrophoneSubscriber.SamplesArray, sent_samples, (int)samples), MicrophoneSubscriber.SamplesFormat);
            sent_samples += (int)samples;

            // if sample have reached the end , returen true to reset  and ResetCapture = true to reset subscribers
            if (sent_samples >= MicrophoneSubscriber.SamplesArray.Length)
            {

                PlayRecording = false;
                return true;
            }
            return false;
        }
        else
        {

            return BasicCapture.UpdateSubscribers();

        }
    }

    void SendSegmentToSubscribers(ArraySegment<float> ReadBuffer , WaveFormat Format)
    {
        for (var i = 0; i < _subscribers.Count; i++)
            _subscribers[i].ReceiveMicrophoneData(ReadBuffer, Format);
    }

    // call to start recording the transmited audio
   public void StartRecording()
    {
        Debug.Log("StartRecording");
        // reset the list that keep track of the recorded samples
        MicrophoneSubscriber.Reset();
        sent_samples = 0;
        // subscribe to the capture system to recive the recorded samples
        comms.MicrophoneCapture.Subscribe(MicrophoneSubscriber);

        // more effective Coroutine (asset from the store) to turn off the recording after a certian period
        Timing.RunCoroutine(StopRecordingAfterSecs(), MEC.Segment.LateUpdate);
    }

    // call to start transmitting the recorded audio 
    public void SendRecording()
    {
        ResetCapture = true;
        PlayRecording = true;

    }

    IEnumerator<float> StopRecordingAfterSecs()
    {

           yield return Timing.WaitForSeconds(RecordingPeriod);
        Debug.Log("StopRecording");
        //create a float array of the recorded samples from the float list
        MicrophoneSubscriber.getSamples();
      comms.MicrophoneCapture.Unsubscribe(MicrophoneSubscriber);

    }

}

using Dissonance.Audio.Capture;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using NAudio.Wave;
using System;

public class MyMicrophoneSubscriber : MonoBehaviour, IMicrophoneSubscriber
{
  public  List<float> Localbuffer=new List<float> ();
    public float[] SamplesArray;
    public WaveFormat SamplesFormat;
    public void ReceiveMicrophoneData(ArraySegment<float> buffer, WaveFormat format)
    {

        Localbuffer .AddRange(buffer.Array) ;
        SamplesFormat = format;

    }

    public void Reset()
    {

        Localbuffer .Clear ();
    }

   public float[] getSamples()
    {
        return SamplesArray = Localbuffer.ToArray();
    }
}

[martindevans]

Localbuffer .AddRange(buffer.Array);

This isn't correct - as I mentioned before the data store in the segment does not occupy the entire Array. You need to look at the buffer.Offset and buffer.Count values and copy out just that slice of the array. It will work most of the time (the microphone tries to fill the entire buffer) but if there's ever a shortage of samples you'll get very nasty sounding audio artefacts.

public float[] getSamples()

It's pretty expensive to copy all this data into a new array. Make sure you don't call getSamples more often than you need to (it look like you're ok on this).

recording_sample_rate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);

This will work for now but it's very brittle - if we change how the sample rate is chosen it'll break. You should look at the WaveFormat object returned by BasicCapture.StartCapture and grab the sample rate off that.

Other than those few things it looks like you're pretty much done, feel free to close this issue if that's the case. If you think I did a good job supporting you please consider giving us a review on the asset store, it really helps us out :)

[Asaadmk]

thanks, i already have a five star review, but i have spiced it up a little, and i will include the asset in the game credit not that it mean much at the beginning but if the game got successful inshallah you might see some results.

[martindevans]

Oh wow, thanks very much 👍

[tamimzoabi]

Hi Please i need your help i need to do the first step of your project : user press a button and start speaking. i configure Dissonance fine . but its start to receive my voice if i press what ever on the scene . i need just to react if i press a Button .

[martindevans]

Hi @tamimzoabi, I've moved your question to a new issue here.