Adding interactive and generative music using webaudio api

[ktatar]

Is your feature request related to a problem? Please describe.

Currently, the audio possibilities are limited in threejs. We (@pparocza and I) have been looking into integration of this following webaudio tools into threejs: https://github.com/pparocza/generative-music-web-audio

Describe the solution you'd like

We want to add new features that allows easy implementation of generative and interactive music in threejs that goes beyond simple audio file playback. We are interested in lightweight audio solutions such as pure synthesis or wavetable usage as in the early games.

Describe alternatives you've considered

An alternative is to code webaudio within each project as opposed to having classes and functions that allows quick prototyping. Another alternative is to use an external library such as tonejs. However, it could be beneficial to have audio synth tools and features directly in threejs to have multi-modal features, such as 3D objects changing properties with audio features.

Additional context

Main question here is, what would be the best way to initiate a conversation on this? It would be nice to figure out an general code structure, maybe a AudioSynth class under Audio? Something that could integrate with current PositionalAudio class and AudioListener class would be beneficial. What would be the best way of merge requests?

To be clear, both @pparocza and I are willing to take the lead and code these features.

[mrdoob]

@ktatar What new classes would you be adding? How would the API look like?

[ktatar]

I think we would add a new AudioSynth class that would be compatible with the current PositionalAudio and AudioListener classes. The audio synth class would be a wrapper of WebAudio api; allowing someone to make a custom synth in a simple manner. Pretty much a simpler version of this repo: https://github.com/pparocza/generative-music-web-audio

It would be cool to have it within threejs, so that we could have visual events happening in conjunction with audio synthesis. Having multimodal / audio-visual events are common in various creative coding practices.

We are more interested in lightweight audio synth approaches, as in the early games, such as using pure synthesis and wavetables. Hence, this new class would not include any audio file playback, which is already included in threejs.

Concretely, the AudioSynth class would be a separate .js file in audio folder. So, it wouldn't affect the rest of threejs code.

What are your thoughts?

[mrdoob]

That sounds good to me, and I'm all about lightweight audio synth approaches too. I would start by adding it in the examples/jsm/audio folder for now.

[ktatar]

Awesome! I will continue this thread once we have sth. substantial to merge.

[pparocza]

Hey there! We've got a few things we'd like to share with if you've got time in the near future? Our fork (and branch) is currently available here:

https://github.com/ktatar/three.js/tree/audiosynth

The modules we created are located in "docs/examples/jsm/audio", and we've incorporated them into some of your html examples in the files "webaudio_synth_interactive_sequencer.html" and "webaudio_synth_timinig.html". Also, the module "SynthPresets.js" contains a series of examples of how our modules can be used to create sounds, and is implemented in the "web_audio_synth_presets.html" example.

These are all rough drafts meant to demonstrate our specific approach to audio synthesis, so we wanted first to see if this would be something you'd be interested in incorporating into the library, and then hopefully begin a discussion about how that would best be accomplished (which is also why everything isn't fully in line with your provided style guide, which we will be happy to follow if we decide to move forward).

Anyway we've had a lot of fun putting all this together, so we're eager to hear your thoughts!

Thanks for your time! Paul

[WestLangley]

test links:

https://raw.githack.com/ktatar/three.js/audiosynth/examples/webaudio_synth_timing.html

https://raw.githack.com/ktatar/three.js/audiosynth/examples/webaudio_synth_interactive_sequencer.html

https://raw.githack.com/ktatar/three.js/audiosynth/examples/web_audio_synth_presets.html