Superpowered Recording delay with synchronization to background music.

[priyankjain12]

Hi, I am using superpowered SDK to record the user's voice while playing the background music. The thing is that when the recording is completed and I merge the background audio and background music with FFMPEG or play them together with the Superpowered SDK, the recording lags behind a certain time from 100ms-400ms. It is not proper with the background music player.

I am using this code to record the user's voice and playing it with the background music. I think that the recording is starting a little bit late but it contains all the input data. So is this issue with the Superpowered recorder or my custom code?

I am using Pixel 3A and one of my clients is using Samsung Galaxy S10 Lite. This problem we are facing on the Samsung Galaxy S10 Lite.

static bool audioProcessing(void *__unused clientdata, short int *audio, int numberOfFrames, int samplerate) {

    if (isPlayerOpened && numPlayersLoadedForRecording != 1)
        numPlayersLoadedForRecording++;

    if (numPlayersLoadedForRecording == 1 && isThereMessageForRecordingPlayer) {
        isThereMessageForRecordingPlayer = false;
        //Check if recording is pause or not, by default it will be false so that first time the player will
        //Call the play method.
        if (isRecordingPause)
            player->pause();
        else
            player->play();
    }

    //A method is setting boolean to determine if recording is pause or not.
    //If recording is pause then return false to disable output.
    if (isRecordingPause || numPlayersLoadedForRecording != 1)
        return false;

    if (echo->enabled)
        echo->samplerate = (unsigned int) samplerate;
    player->outputSamplerate = (unsigned int) samplerate;

    float floatBuffer[numberOfFrames * 2];

    Superpowered::ShortIntToFloat(audio, floatBuffer, (unsigned int) numberOfFrames);

    //Recording only user voice
    recorder->recordInterleaved(floatBuffer, (unsigned int) numberOfFrames);

    //Increasing user's voice volume
    Superpowered::Volume(floatBuffer, floatBuffer, currentVoiceFeedbackVolume,
                         currentVoiceFeedbackVolume,
                         (unsigned int) numberOfFrames);

    //If echo is enabled than processing it.
    if (echo->enabled)
        echo->process(floatBuffer, floatBuffer, (unsigned int) numberOfFrames);

    //Here getting audio from the background music mixing it with the voice buffer so that user
    //can hear the live voice feedback.
    player->processStereo(floatBuffer, true, (unsigned int) numberOfFrames, currentMusicVolume);

    //Processing with the equalizer
    if (bandEQ != nullptr)
        bandEQ->process(floatBuffer, floatBuffer, numberOfFrames);

    //Returning the audio.
    Superpowered::FloatToShortInt(floatBuffer, audio, (unsigned int) numberOfFrames);
    return true;
}

//This method initializes the player and echo
static void initializePlayer(JNIEnv *env, jobject __unused thiz, jstring back_ground_music_path,
                             jint sample_rate_for_native) {

    echo = new Superpowered::Echo((unsigned int) sample_rate_for_native);
    echo->enabled = false;
    echo->wet = 0.5;
    echo->dry = 0.5;

    // creating the player
    player = new Superpowered::AdvancedAudioPlayer((unsigned int) sample_rate_for_native, 0);
    const char *str = env->GetStringUTFChars(back_ground_music_path, 0);
    //Setting this to pause and opening it and again setting the position to 0 to make sure that everything start from 0.
    player->pause();
    player->open(str);
    player->setPosition(0, true, false);
    env->ReleaseStringUTFChars(back_ground_music_path, str);
}

// This method start's the SuperpoweredAndroidAudioIO and initialize the Superpowered Recorder.
static void startRecording(JNIEnv *__unused env, jobject  __unused obj, jint samplerate, jint buffersize,
               jint recordingFileDescriptor) {
    recorder = new Superpowered::Recorder(NULL);
    recorder->preparefd(
            recordingFileDescriptor,
            0,
            (unsigned int) samplerate,
            false,
            1
    );
    isThereMessageForRecordingPlayer = true;
    audioIO = new SuperpoweredAndroidAudioIO(
            samplerate,      // native sampe rate
            buffersize,      // native buffer size
            true,            // enableInput
            true,           // enableOutput
            audioProcessing, // process callback function
            NULL,      // clientData
            -1,
            SL_ANDROID_STREAM_MEDIA
    );
}

//Method to check if background music player is opened or not.
static jboolean isBackGroundPlayerLoaded(JNIEnv *__unused env,
                                         jobject __unused thiz) {
    if (player != nullptr) {
        if (player->getLatestEvent() == Superpowered::PlayerEvent_Opened) {
            isPlayerOpened = true;
            return true;
        } else {
            return false;
        }
    } else {
        return false;
    }
}

// StopAudio - Stop audio engine and free audio buffer.
static void stopRecording(JNIEnv *__unused env, jobject __unused obj) {

    if (audioIO != nullptr) {
        audioIO->stop();
        delete audioIO;
        audioIO = nullptr;
    }

    if (player != nullptr) {
        delete player;
        player = nullptr;
    }

    if (recorder != nullptr) {
        recorder->stop();
        while (!recorder->isFinished()) usleep(100000);
        delete recorder;
        recorder = nullptr;
    }

    if (echo != nullptr) {
        delete echo;
        echo = nullptr;
    }

    if (bandEQ != nullptr) {
        delete bandEQ;
        bandEQ = nullptr;
    }

    numPlayersLoadedForRecording = 0;
    isRecordingPause = false;
    isThereMessageForRecordingPlayer = false;
}

As you can see from the code the player's play and pause event are happening inside to audio processing callback. So is this issue with Superpowered Recorder or an issue of Thread synchronization or anything else?

[gaborszanto]

The latency comes from several factors:

• The UI path from button push until startRecording is called.
• The recoder and audio I/O is created when startRecording is called, but they need some time to "heat up".
• And finally, the famous 10 millisecond problem of Android: https://superpowered.com/latency

[priyankjain12]

Sorry for the late reply @gaborszanto as we were investigating the issue on several devices. The issue was found to be on our side. The thing is that we were using a screen recorder to record the entire Superpowered SDK operation with our app. And it was working well without any issues on all devices except Samsung Galaxy S10 Lite. So once we disabled the screen recorder and checked the entire flow and it indeed worked without any latency issue and it's amazing. So thank you for that.

I have just one question though, as you said

The recoder and audio I/O is created when startRecording is called, but they need some time to "heat up".

So I have checked my code and I will make few changes to it like,

static bool audioProcessing(void *__unused clientdata, short int *audio, int numberOfFrames, int samplerate) {
         if (!shouldStartRecording)
             return false;

        if (isPlayerOpened && numPlayersLoadedForRecording != 1)
              numPlayersLoadedForRecording++;
       //Rest code same as above
}

static void initializePlayer(JNIEnv *env, jobject __unused thiz, jstring back_ground_music_path,
                             jint sample_rate_for_native,jint buffersize, jint recordingFileDescriptor) {

       //Same as above only added this lines from the startRecording method.
    recorder = new Superpowered::Recorder(NULL);
    recorder->preparefd(
            recordingFileDescriptor,
            0,
            (unsigned int) sample_rate_for_native,
            false,
            1
    );
    isThereMessageForRecordingPlayer = true;
    audioIO = new SuperpoweredAndroidAudioIO(
            sample_rate_for_native,      // native sampe rate
            buffersize,      // native buffer size
            true,            // enableInput
            true,           // enableOutput
            audioProcessing, // process callback function
            NULL,      // clientData
            -1,
            SL_ANDROID_STREAM_MEDIA
    );

}

static void startRecording(JNIEnv *__unused env, jobject  __unused obj){
   shouldStartRecording = true;
}

So as you can see I have moved the recorder and audio IO object creation from start recording to initialize player method and inside audio IO I am using a boolean shouldStartRecording to manage the callback only if start recording is set to true.

So do you think it's a good thing to manage this with a boolean or it will increase the performance or anything like that?

[gaborszanto]

Yes, this looks much faster.

[priyankjain12]

Thanks for the info. You can close this now as the issue was on the device.

[samyakkkk]

@gaborszanto we have the below processing callback on iOS.

bool processRecorderAudioIO(void *clientdata, float *input, float *output, unsigned int numberOfFrames, unsigned int samplerate, uint64_t hostTime) {
   ///some code
        if(isPlayerOpen && isRecording && player->isPlaying()){
            recorder->recordInterleaved(input, numberOfFrames);
        }
        if (output != nullptr && player) {
            player->processStereo(output, false, numberOfFrames);
        }
        if (player) {
            player->processStereo(input, true, numberOfFrames);
        }
  ///some code

}

We are also processing the user's vocals so they have feedback on their voice when using headphones. But when using device speaker's this is distorting the player output.

The reason I believe is it creates a loop where player audio goes into the microphone and is played out again with doesn't happen when using headphones.

Does that makes sense? If yes, is there any condition I can add to ensure that vocal is processed in player only when headphones are attached?

Our app is on Flutter and we are using dart:ffi.

cc @agent515