Time Stretching with Oboe and SoundTouch

[stephandue]

Hello,

I am trying to build an Android Mulitrack Player which supports Time Stretching and Pitch Shifting. In order to play multiple audio tracks simultaneously I chose Oboe and started my project based on the DrumThumper Example. Then I added the SoundTouch Library to my Project and added it to the onAudioReady callback:

`DataCallbackResult SimpleMultiPlayer::MyDataCallback::onAudioReady(AudioStream oboeStream, void audioData, int32_t numFrames) {

auto result = oboeStream->getXRunCount();
if (result) { // Check if the result is successful
    int32_t currentXRunCount = result.value();
    if (currentXRunCount != mPreviousXRunCount) {
        LOGD("oboeStream->getXRunCount(): %d", currentXRunCount);
        mPreviousXRunCount = currentXRunCount;
    }
}

StreamState streamState = oboeStream->getState();
if (streamState != StreamState::Open && streamState != StreamState::Started) {
    __android_log_print(ANDROID_LOG_ERROR, TAG, "  streamState:%d", streamState);
}
if (streamState == StreamState::Disconnected) {
    __android_log_print(ANDROID_LOG_ERROR, TAG, "  streamState::Disconnected");
}

memset(audioData, 0, static_cast<size_t>(numFrames) * static_cast<size_t>
        (mParent->mChannelCount) * sizeof(float));

// OneShotSampleSource* sources = mSampleSources.get();
for(int32_t index = 0; index < mParent->mNumSampleBuffers; index++) {
    if (mParent->mSampleSources[index]->isPlaying()) {
        mParent->mSampleSources[index]->mixAudio((float*)audioData, mParent->mChannelCount,
                                                 numFrames);
    }
}

if (mParent->mSampleSources[0]->isPlaying()) {
    // Process the mixed audio data with SoundTouch
    float *floatAudioData = static_cast<float*>(audioData);
    // Feed the mixed audio data into SoundTouch
    mParent->mSoundTouch.putSamples(floatAudioData, numFrames);
    // Clear the buffer to ensure no residual data is present
    memset(floatAudioData, 0, numFrames * mParent->mChannelCount * sizeof(float));
    // Retrieve processed samples from SoundTouch
    mParent->mSoundTouch.receiveSamples(floatAudioData, numFrames);
}

return DataCallbackResult::Continue;

}`

First I had lots of glitches as soon as I changed the tempo for mSoundTouch to anything but 1.0:

mSoundTouch.setTempo(tempo);

After I have changed the PerformanceMode of the Oboe AudioStreamBuilder from LowLatency to PowerSaving:

builder.setPerformanceMode(PerformanceMode::PowerSaving);

And increased the Buffersize I was able to change the tempo below 1.0 without any glitches. But I still face lots of glitches when setting the tempo to above 1.0.

I have the feeling it has to do with the tempo changing the position of the buffer and the total buffesize instead of the processing time SoundTouch needs.

Here is the whole example Project: https://github.com/stephandue/multitrackplayer

It would be awesome if somebody could give me any hints. I´m also willing to get some paid consultation if anybody can help.

Many Thanks Stephan

[philburk]

Real-time time-stretching is not possible. You will need to record the audio, stop recording, time-stretch it, then play it back.

If you are doing pitch-shifting using an FFT then I recommend setting the buffer capacity very high because the FFT is very bursty. You may also consider sending the data to a worker thread through a FIFO.

This advice on signal processing is outside the scope of Oboe so I am closing this.