Add Audio playback quantisation support

[EMBYRDEV]

Describe the project you are working on

A first person shooter with a strong empasis on dynamic music.

Describe the problem or limitation you are having in your project

Unlike most dyanamic music systems, the workflow we are currently using in SPRAWL via Unreal's Quartz Subsystem allows us to compose the song via user input and triggers, queing up the next measure of music to play with sample accuracy.

We can write a systme on top of this that allows us to attach some metadata to a clip of music such as BPM and then inside of quartz we can set up quantisation callbacks that allow us to queue up a sound being played on the next beat/bar. This gives time for the audio thread to collect these playback requests and queue them up allowing for sample accurate timing

I have attached a video below showing the system in action as a usecase for this API addition. https://www.youtube.com/watch?v=the7TL0tq10

Describe the feature / enhancement and how it helps to overcome the problem or limitation

We hope to be able to migrate to Godot for future projects but we see this music system as pretty core to the games we make. We have read up on syncing up audio in Godot and it gets us about halfway to what we'd require to emulate this system.

My proposal is broken up into two seperate but closely reated features:

1. Allow us to queue sounds to be played sometime in the future, the benefit to this would be giving the audio thread time to catch up to these requests and start mixing the sound into the correct buffers in time for playback.
2. Allow us to subscribe to callbacks to be fired on "quantization boundaries". We would be able to tell the audio clock what the current beats per minute is and get a callback every beat which would allow us to request a music change for the next beat.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Here is a rough mockup of what I envision interacting with this API would look like. I am not a professional user of Godot so this may not be the best way it could be implemented, it's meerly a simplified version of what the SPRAWL C++ source code looks like.

var music_clock = null
var current_music_section = null
var queued_music_section = null

func _enter_tree():
    music_clock = AudioServer.create_quantization_clock()

func queue_music_section(section):
    queued_music_section = section

    if current_music_section == null: # play immediately if nothing is playing
        _play_music_section(section)
    else: # wait for right time to play
        music_clock.add_callback(AudioServer.QuantizationEvents.BAR, self, _quantization_bar_callback)

func _play_music_section(section):
    if section == null:
        current_music_section = null
        music_clock.pause()
        return

    # request AudioPlayer to play the sound at the next BAR boundary
    $AudioPlayer.play_quantized(current_music_section, music_clock, AudioServer.QuantizationEvents.BAR, _sound_played_callback) 

func _sound_played_callback(): # update clock on any BPM changes as a result.
    music_clock.set_beats_per_minute(current_music_section.BPM)

func _quantization_bar_callback(quantization_type, current_bar, current_beat, beat_fraction):
    if current_music_section == null:
        return

    if current_bar == current_music_section.bar_count:
        if queued_music_section != null:
            _play_music_section(queued_music_section)
        else:
            _play_music_section(current_music_section)

        music_clock.clear_callbacks()

If this enhancement will not be used often, can it be worked around with a few lines of script?

I dont believe so. I also believe this will be very useful to developers. Almost every developer I've shown the inner workings of SPRAWL to has asked me how we managed to do the music system and when we showed them how to work with the Unreal Quartz Subsystem they have looked to incorporate that in their own.

Is there a reason why this should be core and not an add-on in the asset library?

It's pretty core to how the audio engine works, I'm not sure that this could be done in a plugin.

[Calinou]

Allow us to queue sounds to be played sometime in the future, the benefit to this would be giving the audio thread time to catch up to these requests and start mixing the sound into the correct buffers in time for playback.

This is being tracked in https://github.com/godotengine/godot-proposals/issues/1151.

[fire]

See also https://github.com/godotengine/godot-proposals/issues/3394 for procedural music.

[reduz]

There is a work that was done for a past GSOC that allows this in a very nicely done way, but was not merged yet. I hope it will be able to be merged before 4.0.

[reduz]

The other feature that is requested a lot (which is what I misunderstood this for) is the ability to trigger sample playback start at an exact frame offset internal, to allow syncinc events with music properly. There are many ideas on how to do this.

[lemilonkh]

This might be doable using the beat/ bar signals from my recent PR that I added to AudioStreamPlayer[2D/3D]. Would be good to know if it's precise enough for this use case.
For convenience and to not disturb the timing of the audio thread the signals are triggered on the main thread, so it's not exactly when the audio frame is processed, but it would be in the same game frame.