search

petersalomonsen / javascriptmusic

A.K.A. WebAssembly Music. Live coding music and synthesis in Javascript / AssemblyScript (WebAssembly)
https://petersalomonsen.com
GNU General Public License v3.0
380 stars 32 forks source link

Writing VST and native Audio plugins in AssemblyScript? #50

Closed Catsvilles closed 3 years ago

Catsvilles commented 3 years ago

Hello Peter, yes, this maybe is not completely related to this repo but I think you are the best person to ask about this. :)

A while ago I read your article about converting WASM to C. And this got me thinking if writing a native audio application would be possible with AssemblyScript at all? While doing a bit of research (not having practical experiments though), here are my thoughts: Maybe we could somehow use the converted C code to build the plugins? For example, this compiles PureData patches to C. "The C source can then be used to build VST plugins or integrate with games engines like Unity, Unreal or Wvise".

Another theory of mine, we could write our audio app/synth in AssemblyScript and use the WASM binary files with node.js, or other server environments (Python) to i/o the audio. I'm sure there are also ways to build independent UIs either with Web technologies or some native toolkits. Also, the easiest and most obvious way is using Electrons.js but from what I understand bringing the whole Chromium engine will influence the performance drastically.

These are just theories and I would love to hear your theoretical opinions on this too, pretty much sure some other people will also find this useful. Would be cool to be able to write cross-platform apps in Typescript for both Web and Native, Desktop. :)

petersalomonsen commented 3 years ago

Yes, indeed this is possible.

Actually if using the AssemblyScript midisynth ( https://github.com/petersalomonsen/javascriptmusic/blob/master/wasmaudioworklet/synth1/assembly/midi/midisynth.ts#L208 ) you have methods like shortMessage which accepts regular midi note on/note off, control changes etc. And there's a fillSampleBuffer method that can be used for each render callback (will fill 128 frames of audio data).

The exported wasm files from the webassembly music web app already expose these methods and so they can be used directly in a VST or audiounit by converting them to c (using wasm2c). You could also use them from webassembly runtimes like nodejs or via python like you say. @vshymanskyy has already done this in https://github.com/wasm3/pywasm3 (not with the midisynth though, but the first gen pattern sequencer version. But he's also testing the midisynth).

So this is absolutely possible even today :)

Catsvilles commented 3 years ago

@petersalomonsen Thank you for the answer! I will be digging and exploring more, who knows, maybe eventually I will come up with some tool or library to really making cross-platform audio plugins with AssemblyScript. Cheers to you Peter, I'm following your work very tight and learning a lot from it, so thank you for this too! :)