Allow audio generation inside an extended AudioStreamGenerator resource (rather than a node)

[yankscally]

Describe the project you are working on

custom audio generation

Describe the problem or limitation you are having in your project

after proposing #9777 - I tried my best to make a audio generator that DOES NOT extend from Node or AudioStreamPlayer. Rather the resource should have everything built in so when I drop it into AudioStreamPlayer.stream and play() is called, it plays the stream without no extra script for the player. If I did it using a player script, I would need 3 identical scripts for each player node to handle the custom stream.

I want to make this script:

extends AudioStreamGenerator
class_name AudioStreamNoise

@export_enum("white", "pink", "brown") var noise_type: int

var frames_size = 4096 # seconds
@export var frames = []
var playback : AudioStreamPlayback
func _init():
    print("init")
    fill_frames(noise_type)
    _instantiate_playback()

func _instantiate_playback():
    for frame in frames:
        var audio_frame = Vector2(-frame,frame)
        playback.push_frame(audio_frame)
    return playback
    # really not sure how this works.

I don't see how I'm supposed to get this to work without making a script for AudioStreamPlayer that gets the playback and pushes the frames from this custom resource.

I tried overriding _instantiate_playback() but that doesn't seem to be well documented on how to use it either.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Being able to decide how playback works from the AudioStreamGenerator script itself is the most logical way to do this.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I don't know - because I'm not even sure what is really going on. Hopefully I'm just wrong or missed something.

If this enhancement will not be used often, can it be worked around with a few lines of script?

That's exactly what I'm trying to do.

Is there a reason why this should be core and not an add-on in the asset library?

it's already there, just needs some work (I think, not sure yet)

[AThousandShips]

With GDExtension you can just use AudioStreamPlayback._mix

[yankscally]

With GDExtension you can just use AudioStreamPlayback._mix Unfortunately I don't know any cpp and I don't think I will be able to learn for a while, but GDExtension is very promising for lots of audio things. Trying to get the best out of what I can use in pure GDScript.

I ended up doing something else here that works, but with AudioStreamWAV. It's not even close to what I was trying to do in #9777 or with AudioStreamGenerator.

Using AudioStreamWAV feels counterintuitive, especially because the last function requires me to mess with bytes which is not very user friendly (the last function)... and its also loaded into memory, looped, and played back.

I am really coming around to the idea of proposing ready-made generators like Unreal Metasounds (sine, saw, square, triangle, noise) to expand on #9777. A Curve2D resource could also be used to make different waveshapes, granted you remove the bake resolution before processing. I would call it "AudioStreamOscillator"

extends AudioStreamWAV
class_name AudioStreamNoise

@export_enum("white", "pink", "brown") var noise_type: int : set = bake_audio

var frames_size = 44100 # the amount of frames it takes for noise not sound repeated.. 
var audio_frames = []
var playback: AudioStreamPlayback
var is_stereo

func _init():
    is_stereo = true
    format = FORMAT_16_BITS
    loop_end = frames_size 
    loop_mode = LOOP_FORWARD

func bake_audio(type):
    audio_frames = fill_frames(type)
    data = convert_frames_to_pcm(audio_frames, stereo)

func fill_frames(noise_type):
    var frames = []
    var frames_available = frames_size

    match noise_type:
        0: # White noise
            ...
        1: # Pink noise
            ...
        2: # Brown noise
            ... 

    return frames

func convert_frames_to_pcm(frames, is_stereo):
    var bytes = PackedByteArray()
    var sample_count = frames.size()
    var channels = 2
    for frame in frames:
        var left_sample = int(frame.x * 32767.0)
        bytes.push_back(left_sample & 0xFF)
        bytes.push_back((left_sample >> 8) & 0xFF)
        if is_stereo:
            var right_sample = int(frame.y * 32767.0)
            bytes.push_back(right_sample & 0xFF)
            bytes.push_back((right_sample >> 8) & 0xFF)
    return bytes

[AThousandShips]

I am really coming around to the idea of proposing ready-made generators like Unreal Metasound

See:

• https://github.com/godotengine/godot-proposals/issues/3394