Does Aura support time stretching?

[mikaib]

Hi!

Is it possible to change the audio speed using aura (specifically without changing pitch)?

With kind regards,

[MoritzBrueckner]

Hi, at the moment there is no support for time stretching in Aura out of the box. You can of course write your own DSP effect like you wrote on the Kha Discord so it's still technically possible. You mentioned that you didn't succeed doing this, was this because of Aura (e.g. you didn't understand how to adapt your solution to Aura) or because of the algorithm/maths? Feel free to ask your questions here (or in a discussion), I'm happy to help (if I'm able to do so).

What's your use case if I may ask? There are various time stretching algorithms which not only have a different performance but also have varying quality for different kinds of sounds.

I'm not an expert on DSP (quite the opposite actually), but granular synthesis sounds like a simple algorithm that still gives reasonable results in many cases. Maybe this is a good starting point?

[mikaib]

Thank you for your quick response!

I'd like and attempt to write a rhythm game using haxe/kha, and I'd like to add modifiers to increase speed, which I'd like to do without affecting pitch. I'm pretty new to Aura, which is one of the current factors as to why I struggle a bit. But mainly finding resources/understanding the math behind the required algorithms is currently my biggest problem.

With kind regards, http80

EDIT: Thank you for your help!

[MoritzBrueckner]

If you only want to increase speed as you wrote it's probably rather straightforward to implement, but if you also want to decrease the speed I think you might need to two different approaches for each direction if you want to implement this as a DSP effect. The reason for this is that in each "audio frame", Aura requests a certain amount of samples from each playing channel (currently that amount is hard-coded to 512 but this might change in the future) and only afterwards executes all DSP effects on that channel. This means that the way in which Aura works currently does not suit DSP effects that need to change the playback speed.

But there are workarounds:

• If you want to speed up the playback (= you want to fit more samples into the same amount of time), you can probably call the channel's nextSamples() function with the extra samples you need within the effect's process function. This will also advance the read pointer in the audio source by the amount of samples you specified, so the next call of process() will receive parts of the audio signal starting from the correct position. For example, if you want to process 1024 samples for each audio frame with 512 samples, you would first process the 512 samples Aura passes to process() and then call process again with a new batch of 512 samples from another nextSamples() call. In the next frame, Aura already advanced the read pointer by 1024 samples then.
• Slowing down the playback is more tricky in theory, as process() gets more samples than you actually need. You can't just store them temporarily in a buffer, because the buffer would continuously need to grow as the playback lags behind the audio source further and further. However, I think that there is a really simple workaround for this: You can change the channel's pitch value so that you get a slower sound (which also has lower pitch) which you then resample (there's a helper utility for this) to essentially undo the pitch (with some minor quality losses, I guess). This way Aura only advances a smaller amount of source samples per frame that it would otherwise.
• If you need more flexibility, you could also implement a custom audio channel where you can directly control the playback speed. The downside of this is that you need to dig deeper into Aura internals and you also might need to expose some of them to the code outside of the library.

[mikaib]

Hi!

I managed to implement the slowdown, however I am still confused on the speedup though. Do I just make the channel one of the arguments, then make nextSamples a public function, and call it every process tick? (to advance the read pointer at twice the rate?)

Thanks in advance!

[MoritzBrueckner]

Hi, sorry for my late answer, I fell ill a few days ago and so far didn't have a clear head to actually answer. Depending on how fast I get well again my next answers might also take a few days.

Do I just make the channel one of the arguments

Yes, DSP effects currently can't access the channel they belong to so you need to pass the channel as a constructor parameter, for example.

Implementing a channel attribute for all DSP effects would be easy of course, but I'm undecided about whether I want to add it since DSP effects should be mostly independent of their channels. If effects were able to control their channels this would be kind of an inversion of responsibility.

then make nextSamples a public function

If you don't want to modify Aura you can also use @:privateAccess channel.nextSamples(buffer, sampleRate) instead of making that function public. It's of course somehat hacky, but after all, the entire solution for this DSP effect is somewhat hacky :)

and call it every process tick? (to advance the read pointer at twice the rate?)

What I had in mind is something like this (I now realize that I didn't think this quite through initially, so it's even more hacky and ugly):

// In your DSP effect:

function process(buffer: AudioBuffer) {
    processPrivate(buffer);

    final previousChannelLength = buffer.channelLength;
    buffer.channelLength = /* Whatever amount of additional samples you need, must not be larger than the previous amount */;

    @:privateAccess channel.nextSamples(buffer, Aura.sampleRate); // Request a second batch of samples
    processPrivate(buffer);

    // If you need more than twice the amount of samples per frame
    // because you want to more than double the playback speed,
    // you either need to allocate a larger buffer for calling nextSamples()
    // or you need to call nextSamples() more often until you processed all samples you need.

    buffer.channelLength = previousChannelLength; // Restore length for this audio buffer so that subsequent effects and audio frames aren't cut off
}

function processPrivate(buffer: AudioBuffer) {
    // Your actual time-stretching code
}

What I didn't show in the above example is that you likely need to allocate a second audio buffer because the call to nextSamples() would override the results from the first processPrivate() call. Depending on how you implement this, you might not need to temporarily change buffer.channelLength. If you don't want to call your process() function manually with an arbitrary amount of samples, you can allocate the second buffer with a fixed size in the constructor of your DSP effect, the buffer channel length used by Aura can be found here in case you need it. If you want the DSP effect to be more dynamic (i.e. you want to call process() manually somewhere else and the second buffer shouldn't have a fixed size), you should rather use Aura's BufferCache and request a cached buffer in the process function, similar to how it's done here or with p_samplesBuffer in Aura.hx.

This got a bit more complicated than I initially anticipated, if you have any ideas on how to improve Aura please let me know :)

[MoritzBrueckner]

I guess something like this would be a better solution than what I proposed above (untested and only works for speedup, not for slowdown):

class TimeStretcher extends DSP {
    var channel: BaseChannel;
    var speedFactor: Float;
    var p_additionalBuffer = new Pointer<AudioBuffer>(null);

    public function new(channel: BaseChannel, speedFactor: Float) {
        this.channel = channel;
        setSpeedFactor(speedFactor);
    }

    public function setSpeedFactor(speedFactor: Float) {
        final numAdditionalSamples = Aura.BLOCK_CHANNEL_SIZE * speedFactor - Aura.BLOCK_CHANNEL_SIZE;

        if (!BufferCache.getBuffer(TAudioBuffer, p_additionalBuffer, Aura.NUM_OUTPUT_CHANNELS, numAdditionalSamples)) {
            trace("Could not allocate buffer for TimeStretcher DSP");
            return;
        }

        this.speedFactor = speedFactor;
    }

    function process(buffer: AudioBuffer) {
        final additionalBuffer = p_additionalBuffer.get();

        // Buffer allocation failed
        if (additionalBuffer == null) {
            return;
        }

        final samplesWritten = processPrivate(buffer, buffer, 0);

        @:privateAccess channel.nextSamples(additionalBuffer, Aura.sampleRate);

        processPrivate(additionalBuffer, buffer, samplesWritten);
    }

    function processPrivate(inputBuffer: AudioBuffer, outputBuffer: AudioBuffer, outputOffset: Int): Int {
        // Your time-stretching code. Takes the samples from inputBuffer and
        // copies them into output buffer, starting at the per-channel sample offset outputOffset.
        // This function returns the amount of samples written
    }
}

However, it does not work if you want to call process() manually with custom buffer sizes as it expects the size of the buffer passed to process() to be equal to Aura.BLOCK_CHANNEL_SIZE.

Also note that this isn't really thread safe, since both the audio thread and the main thread might access p_additionalBuffer at the same time. For this you can use the Message system, see this and this for some usage examples. sendMessage() is supposed to be called from the main thread (e.g. for sending the additional buffer to the audio thread), and the audio thread reacts to the message in the overridden parseMessage() function. I'm not completely sure whether sending the buffer to the audio thread doesn't create a memory leak (garbage collection is disabled in the audio thread on static targets that actually have a dedicated audio thread), so better test this. You should definitely avoid allocating memory in the audio thread as this could cause audible clicks and you would need to somehow free the memory yourself, which is difficult with Haxe.

Another advantage of the message system, by the way, is that messages are handled once per audio frame before all the nextSamples() and process() functions are called. This way, a value coming from the main thread doesn't change while the audio thread calculates the output values.