Procedural audio support

[JayPavlina]

I'd like to request write access to the sound buffer so that we can programmatically generate audio in real-time. I mean something like Unity's OnAudioFilterRead method, although it shouldn't actually be an audio filter like it is for Unity because that makes no sense. I don't know a lot about Godot's API, but it could be implemented as a subclass of AudioStream, perhaps called AudioStreamProcedural.

Edit: I found a page in the docs that explains how this could be done

[LinuxUserGD]

It's possible to stream audio as PoolByteArray (data property). Don't know how to edit it.

[marcelofg55]

It is possible to access to the audio buffer, I'm using this on a module, in case it helps you: https://github.com/oamldev/oamlGodotModule

[nobuyukinyuu]

Being able to do this without needing to recompile godot with a custom module would be great. Not just for procedural audio, but also for supporting more audio resource formats...

[hungrymonkey]

@JayPavlina dont forget about a deconstructor

https://github.com/godotengine/godot-docs/issues/1120

I am little bit overwhelm to add it

[hungrymonkey]

btw, if you want a good solution, you should look at the resource import system.

Within the resource import system, you can attach scripts to the modules within the editor

I would try to document but I kinda have a backlog of documentation i have to write.

[NoodleSushi]

I hope there there will be procedural audio support for C# at least, I'm not a fan of using C++

[hungrymonkey]

@NoodleSushi I am not sure how to bind gdscript to c#.

I believe the problem implementation is to create a parent class with the procedural script and bind it to bind_virtual_class.

i am not sure because I never used it before.

I might consider implementing it but I am not sure how the language should look. There isnt a complete design guideline in this thread

[ca3games]

Woun't an easy solution is to support MIDI and being able to play a MIDI file made with code?

[LinuxUserGD]

@ca3games https://bitbucket.org/arlez80/godot-midi-player/src/master/

[DasAmpharos]

+1 for this. I would like to write a GameBoy emulator in gdscript just because and I need to be able to synthesize audio in order to emulate sound.

[vnen]

BTW, MIDI is another beast entirely, but it's being worked on as one of the Google Summer of Code projects.

[follower]

This appears to be another option for procedural audio:

...complete audio solution for Godot 3.1, making procedural sound and adaptive/procedural music possible with a few nodes and a couple lines of code.

• https://github.com/kyzfrintin/Godot-Mixing-Desk

Haven't tried it out yet but has some links to some demonstration videos.

[nobuyukinyuu]

That solution is purely for adaptive audio using sound clips. The reason for proposing access to the audio buffer is to generate new sounds at arbitrary resolution, not just to mix existing ones at the resolution of the game loop's update rate.

[follower]

As an update specifically on "programmatically generate audio in real-time" see:

• "Audio Generators" section of "Godot 3.2 will get new audio features".
• Commit: "Added generator audio stream, and spectrum analyzer audio effect" https://github.com/godotengine/godot/commit/e33764744cb2bf72ee77c823c3beeb6dc870d2dc
• Demo: audio/generator/generator_demo.gd

TL;DR: Real-time audio generation has been added to Godot 3.2 by Juan.

[follower]

In addition, I've realised that this issue seems to have attracted people who have interest in three slightly different/overlapping variations of functionality:

1. "programmatically generate audio in real-time"
2. "programmatically generate audio" (without real-time requirement) (also includes "supporting more audio resource formats" where "generate" means "read samples from a file")
3. Programmatically manipulate pre-existing audio (e.g. "Adaptive Audio" or MIDI file generation)

With this in mind, here's a summary of options (of which I'm currently aware) for each concept (perhaps a version of this summary can be added to the Audio Streams tutorial at some point):

1. Programmatically generate audio in real-time

This will be possible in Godot 3.2 via the functionality described in https://github.com/godotengine/godot/issues/18135#issuecomment-481962760.

2. Programmatically generate audio (non real-time)

While investigating the new real-time audio generation functionality I discovered[1] that it's already possible to generate audio in non-real-time but the existing documentation doesn't really make this obvious.

For details on how to implement this, see:

• AudioStreamSample.data
• "Is it possible to make the audio data in gdscript only?"
• "Creating audio samples from PCM data at runtime"
• godot-midi-player/midi/Bank.gd:_read_soundfont_preset_compose_sample() (example implementation reading data from a SoundFont .sf2 file as found in a project mentioned earlier in this issue).

[1] The discovery was driven by the fact that the new functionality seems to explicitly not provide the ability to loop: https://github.com/godotengine/godot/commit/e33764744cb2bf72ee77c823c3beeb6dc870d2dc#diff-ae33cb21fb3292810087f1ff38dae256R157.

3. Programmatically manipulate pre-existing audio

See:

• oamlGodotModule -- "Open Adaptive Music Library module for Godot engine"
• Godot-Mixing-Desk -- makes "procedural sound and adaptive/procedural music possible"
• The godot-midi-player wiki has a linked named "procedural generate music and play" but the page is currently not present. (Presumably it requires directly creating the same objects as would be created when reading a .mid file.) [Edit: You can now see here & here for examples.]

Hope this summary of options has some useful pointers to get everyone on their way. :)

[follower]

Note that according to "Godot Engine - Maintenance release: Godot 3.1.1" release notes:

New audio features for 3.2 were backported.

I just did a test based on https://github.com/godotengine/godot-demo-projects/tree/master/audio/generator and I did indeed hear sound...

So looks like the presents came early this year. :)

Have fun @JayPavlina!

[nobuyukinyuu]

I checked version 3.1.1 on Steam and the new audio features do NOT appear to be backported. Scenes which utilize AudioStreamGenerator will not open. Obviously works on current master. Are the release notes incorrect?

[follower]

This was the version I tested: https://downloads.tuxfamily.org/godotengine/3.1.1/Godot_v3.1.1-stable_osx.64.zip

Has Godot version 3.1.1 been uploaded to Steam yet? (I looked at https://store.steampowered.com/app/404790/Godot_Engine/ and the version there appeared to only be 3.1?)