Make a plugin/addon API

[MainCharacteroftheProblem]

This is a bit of a pipe dream and would require a lot of effort on our part (effort that would be better spent elsewhere). However, I think that if anyone could add this to Beepbox, it would be us.

[AstroScraps]

what is meant by plugin/addon api? like what would fall under a plugin? first thought that comes to mind is a vst, but iirc u guys said that's not really gonna be happening any time soon

[leovoel]

Needs a lot of design work!

There's a bunch of stuff that could be called a plugin. The things I'm aware of that could be dealt with are:

• Audio or song data processors and generators: This is what people are usually thinking of when they say "VST"s or "VSTi"s. Their inputs and outputs are usually audio data, though some can operate instead on MIDI, but in our case we'd maybe not bother with that since it'd require a conversion step. It'd allow people to implement new instrument types and effects. The main API for this would be some way to operate on blocks of audio coming out of the synthesizer. For song data/MIDI I don't actually know what is commonly done, needs research.
• Scripts: I don't know if that's always the name used for this (some would call these "macros" I guess?), but there is usually also a way to automate actions on the editor, and this would be how. The APIs here would generally mirror buttons and other actions available on the editor.

Performance is an important constraint here, and is the main complicating factor here I feel. Not only the plugin can be its own perf can-of-worms but we also have extra data copies and such to worry about on our end. Probably we'd also want to figure out the threading story first before doing anything here. Performance issues also interact badly with my next point...

Security is another important constraint, perhaps more so in our case. Ideally we'd be able to "sandbox" plugins well, but it's difficult to do that while keeping things very fast, which we need for audio.

As for the implementation language we'd open up for plugin authors, it could be JavaScript as the rest of the editor uses, but maybe WebAssembly would be the better option here. We can't load shared libraries (.dll or .so or whatever) in browsers, only in offline versions where we control the executable, so "VST" support can only happen there.

The first actual step here would probably be to look at plugin formats out there and see what we could borrow good ideas from.

[icejix]

Hi everyone, just want to add some of my own ideas, since I've been thinking about a similar thing for a while.

Performance is an important constraint here, and is the main complicating factor here I feel. Not only the plugin can be its own perf can-of-worms but we also have extra data copies and such to worry about on our end. Probably we'd also want to figure out the threading story first before doing anything here. Performance issues also interact badly with my next point...

I'm not sure what the programming guidelines are for UltraBox, but couldn't you just have a text field somewhere on the page, and then eval() or new Function() it? In the case of new Function, you would just call the function for each "frame" of audio. Personally, I want to make some custom audio effects in JS to put on my instruments. I think it would be fun, and allow me to be more original with how my music sounds. In my opinion, you shouldn't overthink it too much, although if VST support is at all possible, it should be added, but not in place of this idea.

Security is another important constraint, perhaps more so in our case. Ideally we'd be able to "sandbox" plugins well, but it's difficult to do that while keeping things very fast, which we need for audio.

An appropriate warning would be good. Since UltraBox hardly stores any cookies, the only real bad things that could happen as a result is that the preferences are reset, or the script spams alert() / something just obnoxious. This would not be a huge issue in my opinion, since you can easily quit a page which is spamming alerts nowadays, and other websites can run arbitrary JS code anyway.

As for the implementation language we'd open up for plugin authors, it could be JavaScript as the rest of the editor uses, but maybe WebAssembly would be the better option here. We can't load shared libraries (.dll or .so or whatever) in browsers, only in offline versions where we control the executable, so "VST" support can only happen there.

If UltraBox / BeepBox is capable of processing audio in JS, and does not need webasm, the plugin system should probably be in JS too I'd say. I'm under the impression more options is usually good, so maybe both? If you're willing to put in the effort of doing all that, that is.

Also, if there were some kind of system to allow for JS-controlled audio, instruments, etc, you could probably make a VST plugin system on top of that, and you wouldn't even need to make any changes to the original source code of UltraBox. Obviously, you'd need to put it in an executable container like Electron first, but then you could write the logic for the input/output of the VST on top of UltraBox's new JS plugin system.

[leovoel]

Hello! It's cool to see other people interested in this stuff :)

I'm not sure what the programming guidelines are for UltraBox, but couldn't you just have a text field somewhere on the page, and then eval() or new Function() it?

You could. new Function is the actual viable option - code executing through eval typically runs very poorly, whereas creating a new function does go through the JIT compiler like the rest of the code (shaktool already uses this technique inside BeepBox to e.g. get rid of ifs in some of the inner loops!).

The thing is that that's too simple.

The simplest API we could offer for instruments is a "buffer" that you have to fill in, and some data indicating the properties of the note that's currently playing, which already is a bit tricky.

For effects it'd also be a buffer, one with the results of instrument synthesis, but then that means we have to customize the effect stack, which is very hardcoded at the moment.

We probably would need some ways of exposing, say, sliders or some other UI things like that. We also of course need to store all of this inside songs somehow, and right now basically nothing on the inside was designed expecting this sort of thing.

An appropriate warning would be good. Since UltraBox hardly stores any cookies, the only real bad things that could happen as a result is that the preferences are reset, or the script spams alert() / something just obnoxious.

Warnings would be a requirement, yeah. As for security of "server-side" components, yeah, we don't have any of that to worry about (though do note that "multiplayer" is also a thing people have thought about - see #124), it's basically "self-XSS" stuff that would be a concern, which I don't really care about, personally. That said, browsers still get exploited in all sorts of ways, so this all does give me a bit of pause.

If UltraBox / BeepBox is capable of processing audio in JS, and does not need webasm, the plugin system should probably be in JS too I'd say. I'm under the impression more options is usually good, so maybe both? If you're willing to put in the effort of doing all that, that is.

I was thinking of WebAssembly mostly just for the more consistent performance, which is better for audio. JavaScript is extremely wobbly performance-wise, and I've personally been investigating moving more of the synthesizer and editor to WebAssembly regardless of this stuff.

For the "type some code in a text input and run it" case, yeah, JS is the only realistic option. Also we could certainly do both, they probably would look very similar API-wise anyway.

I think a VST system would still be better separated from all this, as I know off the top of my head that there's extra information needed there that you wouldn't care about much when experimenting with from-scratch DSP stuff.