GUI Support

[Earlz]

So, I've been trying for the past few days to get any form of GUI working in my VST host (Reaper) using rust-vst2, and I've faced nothing but difficulties. I'm also pretty new to Rust itself, so my inexperience doesn't help.

Anyway, have you had any success prototyping such a thing, and ideas on what to use? I know the readme says Conrod+SDL, and this is the primary thing I've focused on, but I can't help but wonder if there might be an easier and more stable way. The C++ VST kit that everyone, wdl-ol, includes an OpenGL framework with built in easy support for drawing knobs and such. I'd be curious if doing FFI to that might be easier. Do you have any ideas?

[overdrivenpotato]

So if you look at src/editor.rs, it is up to the plugin to create an editor that implements the Editor trait. One of the things that we have to work with is a window pointer, which can be used by the plugin to do something e.g. initialize an SDL2 context. It seems libraries like wdl-ol abstract over more than just the bare VST api, I'm thinking it might be out of scope for this library to do something like that.

There are a few approaches that we can take though, the main thing I'm interested in is seeing if we can get Piston's 2D graphics library to work with an editor. I don't necessarily want to use OpenGL for drawing as it might not work on all systems and we can use a simpler interface for drawing 2D UI elements. If it's easier though I'm good to get just something running with a UI even if it is OpenGL.

We would also need to have the appropriate button controls like knobs and sliders, I'm thinking these can be done in another crate to separate concerns (or we can use conrod!). It also opens up the possibility of code reuse for something such as an Audio Unit rust implementation which needs UI elements as well.

There's a lot of stuff that needs to be done and I'm OK with trying out a bunch of things, just not sure which direction to head in. If you want to take a stab at it, the very first thing would be to see if you can get a graphics context to work with the pointer given in Editor::open. I'll see if I can help out if you do end up making a public fork/repo.

[Earlz]

Yea, I definitely think it'd fall under a completely different crate, but I feel like it could belong to the same project, or at least be related..

I did some investigating in the wdl-ol code, and I think the way they handle it (very thick codebase, so difficult to be sure) is that they basically have a platform dependent native set of graphics code for each platform. This graphics code handles basically the bare minimum however. It exposes a framebuffer, and allows for the normal event setup (ie, keys, mouse, etc). Then, most programs use their LICE abstraction that works directly on the exposed framebuffer for shapes, glyphs, etc. And finally, the programs that care about OpenGL, basically setup OpenGL to render to an invisible hidden window, and manually blit/copy the framebuffer of the OpenGL window to the VST-host window.

I'm not sure that this is the best way to go, but I'm sure there's a reason why they chose this way rather than directly rendering everything using the window handle.

I've been trying to get Editor::open to work properly with SDL, but having very little success thus far.. I intend to keep chasing this though. It'd be awesome if VST work on Rust was actually preferred over C++, since already this is incredibly easy compared to making a VST plugin in C++.. just the GUI bit isn't there, and eventually the DSP other common audio operation helpers

[overdrivenpotato]

Sorry about the delay, holidays and all :tada: (Happy new year!)

Alright well it looks like you're right about the wdl-ol implementation, we can definitely do the same thing and implement platform specific graphics code as I am not aware of a rust library that does this. One of my goals would be to have rust-vst2 work with Linux, OS X and Windows though so I would like to support all three platforms properly. I guess the first step is to decide whether we're going down the route of using graphics as a base or whether we should just create a new library from scratch, your thoughts?

[adamski]

You might want to look at how the JUCE C++ framework deals with different platforms. A rust equivalent to JUCE would be amazing, albeit a very ambitious project!

[suhr]

graphics provides primitives to draw, but you still need a backend for it to initialize an sdl context.

I have a different problem though. I want to implement a microtonal (31 tone) sequencer (like Hex). It will have a complex UI, so I rather should use gtk-rs. But I also want it to be a plugin, to use it in my DAW of choice.

So, how do I use gtk in vst? I heard this is not an easy thing to do, but I also know Carla uses Qt and can be loaded as a plugin.

[sklopi]

Did anyone have luck in initializing a graphics backend from the hwnd handle on windows? I'm currently trying to get this working, but i'm kinda stuck.

[overdrivenpotato]

@suhr I'm not really sure how this should be handled, I do think the best way going forward would be to use some sort of a toolkit but as I don't have experience with either QT or GTK I don't really have any suggestions.

It looks like it boils down to either getting an existing toolkit to use the raw window handle for each platform, or creating a GUI system specific to this library. Personally I'd be interesting in seeing something such as electron being used for the GUI and delegating the implementation/DSP down to rust code, but again I'm not sure where I'd even begin.

[ycros]

I have proof-of-concept working currently on Windows at https://github.com/ycros/vparty using a fork of glutin https://github.com/ycros/glutin/commit/dd0de4444dbcc6c4be86ebf6d3d61a9ffc804ddb - eventually I'd like to get conrod up and running, but this requires either implementing a whole new conrod backend or getting some changes into piston. I know how to get things running on OSX as well, though things are a little trickier there (your pointer may be a Carbon windowRef or an NSWindow depending on 32bit/64bit/your daw). I've been busy with a new job so I haven't had time to come back to working on this.

Basically, for whatever sort of gui lib you want to end up using, you will need to handle spawning your window as a child window under whatever window handle/pointer/thing you get from your host. There's other open source C++ VST wrappers such as wdl-ol that you can peek at to see how they handle gui initialisation.

[Boscop]

I forked glutin, conrod and the piston window stuff and modified it to support multiple child windows in the same thread (required for VST but by default glutin creates a new thread for each window and doesn't allow child windows) and passing the parent window handle all the way through to where the win32 window creation functions are called, you can find the forks on my page, but I ran into some issues like this: https://github.com/PistonDevelopers/piston_window/issues/167 (The problem is piston_window uses its own event loop. I either don't get draw events or no input events.) Right now I'm working on my own lightweight GUI lib on top of glutin, I already have layouting, buttons, knobs, font rendering etc (with batched rendering). but then I got sidetracked working on a game.. I'm getting back to this though. My plan is to create a VST framework on top of this lib that includes GUI support and common DSP routines that are useful and can be composed to create larger plugins, kind of like JUCE but for Rust. I currently have a basic starting point, ported some code from Will Pirkle's book to Rust. If you are interested, we can plan this together on IRC, #rust-music I think Rust has great potential for audio DSP.

Btw, this is an old screenshot of my GUI lib (this example isn't running in a VST but it works in a VST too):

[Boscop]

Another old screenshot, this VST is not using my GUI lib, just my glutin fork.

[suhr]

My plan is to create a VST framework on top of this lib that includes GUI support and common DSP routines that are useful and can be composed to create larger plugins, kind of like JUCE but for Rust.

Then it should also support LV2 and AU.

[Boscop]

@suhr Yeah, but I don't know anything about those formats... So it would be helpful if we can work together. Probably we should design an intermediate API for plugin architectures so that each plugin only has to interface that API and can be compiled to all supported backends. I think that's how WDL-OL does it. Currently, my glutin fork only allows creating child windows on windows. But I abstracted the WindowID into a type, which can be defined in a platform-independent way, so feel free to submit a PR for other platforms. Also I added set_timer() and kill_timer() so that the VST GUI can be drawn at a consistent framerate triggered by the timer, that's how it should be done on other platforms, too. This requires storing a pointer to the plugin in the userdata of the window, this is how I do it on windows. My glutin fork also supports sending a Timer event instead of using a callback for the timer, but you can't poll events out of nothing, something has to trigger it, and if the mouse is not over your VST window, the host will call idle on the plugin only at like 4 fps. That's why I switched to the timer callback, because my spectrum view needed a higher framerate.

[overdrivenpotato]

@Boscop This looks great, I definitely would be interested in discussing this further. Which IRC network is #rust-music on?

[Boscop]

@overdrivenpotato irc.mozilla.org

[suhr]

cast @poidl

[zyvitski]

I think the idea of using CEF would be a good one. Step one would be to get a CEF window running with its event loop running on a separate thread of course. As far as communication between the window and the VST, there would need to be a queue of events being passed from the window handler into the VST. The queue would need to be available to the processing callback and ideally it would be a lock-free SPSC queue. A big question that needs to be asked is wether this library should be responsible for the actual rendering and layout or if it were to only provide the window and the IPC mechanisms needed to communicate with the plugin. I feel that a good starting point would be the latter option and then work on a standardized set of UI elements and such.

[Boscop]

@overdrivenpotato That screenshot looks cool! Can bindgen be used for generating bindings to CEF?

[Boscop]

I found another alternative, these guys are using awesonium for their game/editor GUI: https://www.reddit.com/r/rust_gamedev/comments/5vqlln/shar_one_year_with_rust/ https://youtu.be/OVYQs3KY2EE?t=55 http://www.awesomium.com/ https://github.com/not-fl3/awesomium-rs I think this is a very promising possibility for VST GUIs.

[adamski]

Personally, I don't think HTML is a good fit for these kinds of apps. Although something like React's JSX for declarative UI could work. Audio apps often require lots of custom UI elements, some of which need to be updated at high rates to represent what's going on in the audio. I might have got the wrong end of the stick here though...

[piedoom]

I agree. I think at a very base level, declarative UI is really nice and accessible (like XAML) but I don't think running a web UI for a VST would be the best idea.

[zyvitski]

Has anyone made any progress on getting a GUI setup? I'd be interested to hear how you dealt with the issue of working with the native window handle that's vst hands you

[Boscop]

@zyvitski Here is a description of what I'm doing in my glutin fork: https://github.com/tomaka/winit/issues/159 I haven't updated my forks in months though (I've been busy with work), the conrod fork should be updated to support the glium backend so that one can render stuff with custom shaders... I'm working on a kickdrum plugin right now where I plan to use raymarching to render a GUI with correct shadows on the knobs / glass LEDs and using different noise functions to generate textures like wood for ultimate skeuomorphism...

[zyvitski]

@Boscop Are your changes to glutin only setup to work on windows or have you setup a cross platform solution as well? I have both a Mac and Linux setup and would love to get things working across the board.

[zyvitski]

@Boscop I forked a copy of your glutin fork, I will see what I can do to get it going on OS X. Once I have it setup I will issue a pull request. I think It would be a good idea to try and set up a portable timer to replace the windows timer you used ( can just be a wrapper over the windows one with conditional compilation). Once that is up and going I will give it a go on my linux box.

[zyvitski]

Is anyone familiar with how the cocoa api backend for glutin works? I have never dealt nuch with cocoa. I know that the window handle coming in from the host should be a NSView* (I chekced the vst2 sdk source) but I am not sure what the proper method for working with the NSView is? I tried setting it up where the the view created by glutin would be attached to the incoming view but that just resulted in having two windows.

Is the host handing me a new window to work on or a handle to it's window to attach a new window to?

[Boscop]

I think it's a great idea to have cross platform support, but I've never dealt with OS X so unfortunately I can't help much there. But I know it also has a way to store data in a custom pointer associated with a window (which I use to store the plug-in pointer so that the timer callback can access the plug-in.

The host creates a window with or without decorations (depends on the host) and passes the handle to the plug-in. It has to create a child window without decoration in that parent window.

On windows, all child windows have to live in the same thread because all parent windows live in one thread and child windows have to run in the same thread as their parent. Hence the solution with the event hash map.

To bring winit up to speed with the fork it would require to apply the same changes outlined in that post, but in a cross platform manner. So we need these things as cross platform:

• callback timer on window with callback having access to plugin (e.g. store pointer in window user data)
• a way to create child windows such that they run in the same thread (abstracted window handle as hashmap key for event queues)

On all platforms the window handle is a pointer, so we can just have a WindowId or WindowHandle type that is basically a pointer.. (in my forks I called it WindowId)

Once we have the cross platform windowing going we can also create au plugins... :)

[zyvitski]

After spending an entire 8 hour day trying to model your changes to glutin in the osx backend I finally gave up on it and decided that I am just going to opt for the nuclear option on solving this problem once and for all. I have started work on a native port of vstgui-4 which so far is turning out to not be that bad of a task (time consuming yes, but pretty straight forward). Shouldn't take me too long to get the basic building blocks in place to at least get things moving (without widgets and fancy features). As mentioned above I don't wave a windows box, If anyone would like to jump on board once the skeleton is down and cover the windows backend that would be great ( @Boscop, or anyone else interested).

[adamski]

As a JUCE developer who is watching Rust and hoping to dive in soon, I very much value being able to use the same GUI classes to build plugins as well as standalone apps. It would be great if we could use something like Conrod or Glutin to build plugin GUI's rather than be tied to a VST-specific library.

[Boscop]

I've never worked with vstgui but if it's possible to create professional looking GUIs with it, I'm interested.. But if vstgui works on Mac, we can see how it does the window / event handling there and do the same in glutin / winit.. Then we can use conrod on top, and custom shaders for fancier knobs and custom widgets like spectrum views..
@adamski Great to have a juce developer here too. I hope together we can make Rust more popular for VST development :) Are you working on the juce library or making plugins with it?

[suhr]

I hope the conrod itself won't be mandatory. It's quite complicated.

[adamski]

@Boscop I am mainly writing desktop and mobile audio apps, also with React Native for UI on mobile. I have a couple of plugins in the pipeline. I've made some small contributions to the Juce library, and done some work to make it compatible with React Native.

[zyvitski]

I have done some more digging into how the vsugui sdk handles things for OS X in the backend. From what I can tell the window handle passed in from vst is a NSView* and they create and attach their own undecorated NSView as a subview of the provided NSView I have tried replicating this using glutin but the issue seems to be that the innards of the OS X portion of glutin insist on creating an NSApplication an NSWindow and the necessary NSView (the only part we need. Glutin does this because it needs to setup a full app environment for you, but all we need is event handling and drawing since we are working as a plugin. I Just need to figure out the best course of action for replicating the action from vst with glutin.

For reference, the code from VST that needs to emulated comes from cocoasupport.mm in either vstgui.sf or vstgui4 in the function VSTGUI_NSView_Init.

[Boscop]

In my glutin fork I introduced a member parent: Option<WindowID> for the window builder, so if you do that, then you can check on every step if parent.is_some() and then create the outer stuff only if parent is None.. I'm very busy until tuesday but after that I'm also taking a closer look at vstgui and the way glutin does things on OS X.

[zyvitski]

@Boscop I think part of the issue is that I am still familiarizing myself with the glutin codebase. I am still trying to get a feel for how things work and which portions of the process are necessary for the core functionality of what we may need. I have been using the parent you introduced to try and do just that, but I keep on winding up with two windows.

[monomadic]

Any movement on this? I am stuck in the same position, just spent hours trying to sift through glutins code on mac, it's complex and doesn't lend to just simply providing a child NSView. I looked at VSTGUI and wow, that thing is crazy bad, one thing I've learned out of all this plugin dev is how grateful I am rust-vst2 exists so I don't have to use Steinbergs insanely poor code.

@zyvitski you mentioned you were porting VSTGUI, could you publish your changes? I'd be happy to contrib if you have anything. I've also looked into editing glutin, perhaps we can share findings.

@boscop you seem to be fairly well researched in glutin, can I help you in some way? I am decent with apple's frameworks but somewhat new to rust.

[zyvitski]

@robsaunders I haven't made any headway on it lately. Vstgui4 runs in something around at least 100000 lines of code, I just don't have the time right now to put in on porting it. It's essentially a full graphics library so I have come to the conclusion that it probably would not be best to try and port it over.

[monomadic]

I figured, vstigui is really bad. Just when you said it seemed easy to you thought maybe you knew something I didn't :P VSTGUI 3 is less than a tenth the size, but I'm not 100% sure it even supports cocoa at all (might be dating back to when VST gave you a carbon window handle).

We really need a solution to this as the project isn't going to get far without it. I've already build a bunch of plugins now in rust but have left GUI stuff until now. I'm poking around with using cocoa-rs directly to create and draw on a context, but that doesn't leave much in the way in windows support. I'd prefer not to have to recreate the wheel but I actually don't have a lot of confidence in glutin/pistons architecture, and anyway @Boscop asked them over a year ago about childless window creation support and it has gone ignored, so I don't think they're interested.

I can't even recreate boscops source changes to glutin, looking at his diffs, because the project has changed so much in that time (and I don't think I want to use the version from a year ago). It might be simpler to attack winit and pass that to glutin (WindowBuilder optionally takes a winit object).

Which leaves us with creating something new and perhaps heavily borrowing from those libraries... unless I'm missing something?

[zyvitski]

@robsaunders I think your appraisal of the situation is correct. The issue here is that it will be a huge project to make this happen from scratch. That will require that everyone here is on board with the idea and willing to try and available to put in some time to make it happen. It's just a very big task that we will need to figure out.

[sklopi]

I would actually like to help, but I'm not realy experienced with this gui stuff. But if someony helps me or lets me help, i would be glad if i can contribute. Can test on Linux and Windows.

[monomadic]

For what it's worth, bypassing all of this stuff I can use cocoa-rs to generate a window within bitwig studio pretty easily.

I'm not sure how it behaves on other DAWs yet, and I'm not sure of what limitations I'm about to hit.

Obviously it's mac only, too...

[monomadic]

@zyvitski not sure I agree that it's a super big task, we really just need a better way to generate some kind of window object that is compatible with another library. There's PLENTY we can re-use. If we, for example, made an ACTUAL fork of glutin (vst-glutin? glutin-parentless?), and all just fixed up the current version so that it worked across all platforms, we'd be ok. The glutin objects are the same and can be fed into other connector libraries like piston or whatever sits on top of the regular glutin, and not know the difference. The problem will be divergence of glutin itself after that, so major changes would have to be merged back into the core, but as long as the glutin fork had the SOLE job of supporting childless windows, it would be fine.

[zyvitski]

So here's my game plan: I will open up a new repo and we can get things moving. It's good to know that you can get it going with cocoa.

Like I said, I will init a fresh repo with an empty cargo lib project. Then I will post a link to it here. Gimme around an hour.

[zyvitski]

We can figure out who needs write access and all along the way.

[monomadic]

This is not glutin, above. It's just manually digging into cocoa with cocoa-rs. Have you made any headway yourself? If not it's premature to create any repos just yet. I think @Boscop is probably furthest along, no?

[zyvitski]

@robsaunders I understand. But that's actually better since we probably shouldn't depend on glutin

[monomadic]

If we don't depend on glutin, or something like it, then that IS a large project :P I just need a button or two for now for testing, that's why I did this.

[zyvitski]

We could depend on it if needed. But we need to brainstorm and figure out what the easiest plan of action will be.

[zyvitski]

We just need to figure out how much of the wheel to re-invent

[zyvitski]

Here we go: https://github.com/zyvitski/vstgui-rs

[zyvitski]

I have added @robsaunders and @sklopi as contributors. Anybody else who wants access just let me know.

Make sure to accept the contributor invite links when you get them.