[QUESTION] General "reactive" limitations with REAPER extensions?

Hey Helgo =)

I've been reading through the source for the reaper-rx API to try to get some better understanding of how this kind of architecture/approach would work in REAPER in general.

Specifically, I am interested in building an extension for realtime collaboration on projects (similar to Google Docs collaborative editing). It seems like using an observer/pubsub reactive pattern, where each change event is emitted to peers and they just perform the same event in their own DAW, would be the least complicated way to do it.

Currently the "project history"/"undo history" API is not exposed, and there are no native events emitted you can listen to for changes. (FR here: https://forum.cockos.com/showthread.php?t=248055)

So it seems like a custom rx-ish implementation would be the only way to go.

But:

In reading the source, it looks as though not all REAPER events have change events which you can hook from ControlSurface interface?

IE, I didn't manage to find event names for things like "Inserted/Deleted MIDI notes", "Changed clip length in arranger" etc. So is it only a limited subset of things which you can react to?

https://github.com/helgoboss/reaper-rs/blob/76eb0047c80089d8b1f30de7c62a3e4fef1f4587/main/high/src/change_detection_middleware.rs#L900-L938

I'm wondering whether this is even possible, if that is the case. Pseudo-code, the extension would roughly look something like this

# Could be P2P connections between users, or a self-contained Python websocket server, etc.
from libp2p.network.stream.net_stream_interface import INetStream

PEER_STREAM = INetStream() # Some multi-address here

# When track added in the DAW, broadcast that it's been added to all peers
reaper.track_added.subscribe(lambda track:
   broadcast_track_added(track))

async def broadcast_track_added(track):
  change_event = {
    "type": "TRACK_ADDED",
    "item": track,
  }
  serialized = json.dump(change_event)
  # Send track info and event type to all peers through writeable stream
  await PEER_STREAM.write(serialized)

def handle_broadcast_track_added(track):
  # code here to insert the track in the project
  # when a peer adds a new track + broadcasts it

Also, I know this has technically nothing to do with reaper-rs itself, so I'm more than happy to bugger off and close this issue 😅

Before answering your question, let me just quickly point out that only reaper-low and reaper-medium are intended for public use at the moment. reaper-rx and reaper-high are completely unstable and subject to change.

Now about your question. I think for something like project real-time collaboration (which is an interesting idea!) you would need access to almost all kinds of events that could possibly happen within the REAPER project. The REAPER API exposes quite a lot of event kinds via its control surface API. Most of them are covered by reaper-rs, but some not yet (see the CSURF_EXT_ constants in reaper_plugin.h) and some are maybe undocumented. But I think this is just a fraction of all possible event kinds. I'm by no means an authority here (that would be @justinfrankel) but I doubt that the REAPER devs will ever expose all possible kinds of events via CSURF_EXT_ mechanism. I guess there are too many of them (hundreds I suppose).

I suspect tapping the undo history would be more promising in your case (maybe combined with above events). Probably much easier to add by the devs ... this would be just one simple function pointer dealing with data chunks (strings) instead of 500+ function pointers where each one has another custom-tailored signature. Maybe you can drop Justin a mail to see if he's willing to implement such a hook?

The kind of Rx/p2p code in your code snippet is definitely possible. I'm doing similar things with WebSockets in ReaLearn's projection feature. Just one hint if you want to do it with Rust or C++: Initially I had everything sprinkled with Rx-ish code (also the old C++ version), but in the past weeks I factored all the Rx stuff out of reaper-high. Soon the "inner onion layer" of ReaLearn will also be Rx-free. I love Rx but I found it's a bit harder to use in languages without garbage collector. Still worth it in some cases (e.g. some UI scenarios), but I wouldn't use it all over the place anymore. An event-loop based design appears cleaner and more straight-forward to me now with such languages, especially regarding things like ownership/lifetime and reentry. That's why I introduced MiddlewareControlSurface and ChangeDetectionMiddleware.

Thank you a ton for taking time out of your day to respond! 🙏

Before answering your question, let me just quickly point out that only reaper-low and reaper-medium are intended for public use at the moment. reaper-rx and reaper-high are completely unstable and subject to change.

Yeah, absolutely -- warning heeded. I was just curious how some sort of reactive framework/change detection could be built for REAPER extensions, so peeking under the covers gave me a great place to start googling. (This is how I learned you can use a sort of stub/mock ControlSurface just to hook callbacks, which I believe is what SWS does as well)

I suspect tapping the undo history would be more promising in your case (maybe combined with above events). Probably much easier to add by the devs ... this would be just one simple function pointer dealing with data chunks (strings) instead of 500+ function pointers where each one has another custom-tailored signature. Maybe you can drop Justin a mail to see if he's willing to implement such a hook?

This makes sense and seems the most reasonable, seeing that CSurf were meant to be, I think, actual control surfaces and not really abused in the way a lot of extensions use them. I will maybe write a mail then, hopefully it's not to much a bother, I'm sure he gets many such requests.

I love Rx but I found it's a bit harder to use in languages without garbage collector. ... An event-loop based design appears cleaner and more straight-forward to me now with such languages, especially regarding things like ownership/lifetime and reentry. That's why I introduced MiddlewareControlSurface and ChangeDetectionMiddleware.

I will peek at the source for these! The ChangeDetectionMiddleware lets you register callbacks/handlers to run on change events I assume, sort of like Javascript/Node's EventEmitter interface?

project.on('track_added', (track) => {

})

When digging through the source trying to learn how stuff works, I realized that you've got much, much more plumbing in reaper-rs than other libraries (beyond-reaper, reapy, etc) so probably if I wind up making a serious attempt at this and can solve this issue with getting the change stream/history, I'll just do it in Rust with this.

(Can you cram a webview in as the UI for native Reaper extensions which are dockable, or must you use a specific GFX API?)

I will peek at the source for these! The ChangeDetectionMiddleware lets you register callbacks/handlers to run on change events I assume, sort of like Javascript/Node's EventEmitter interface?

Kind of. Looks like this:

    fn handle_event_internal(&self, event: ControlSurfaceEvent) {
        self.change_detection_middleware.process(event, |e| {
            use ChangeEvent::*;
            match e {
                TrackAdded(_) => println!("track added"),
                TrackRemoved(_) => println!("track removed"),
                _ => {}
            }
        });
    }

When digging through the source trying to learn how stuff works, I realized that you've got much, much more plumbing in reaper-rs than other libraries (beyond-reaper, reapy, etc) so probably if I wind up making a serious attempt at this and can solve this issue with getting the change stream/history, I'll just do it in Rust with this.

I encourage you to do so! Getting used to Rust might take a while but it's a very rewarding time investment in my opinion. If you are okay with garbage collection though, then other languages might be move convenient. I chose Rust because I need to do stuff in real-time threads and I was fed up with C++.

Can you cram a webview in as the UI for native Reaper extensions which are dockable, or must you use a specific GFX API?

Honestly, building the UI can be quite annoying. Some options that I know about:

Boscop's web-view works and I already gave it a shot, but last time I checked, embedding it into a parent window - which would be needed for docking - is not solved yet.
Then there's Cockos SWELL, which basically brings old-school Win32 GUI programming to macOS and Linux and uses native UI components. ReaLearn uses it all the way because of its "tool" character (where typical OS dialogs fit in nicely) and because I wanted it to be lightweight. But it's a pain to work with, especially if you are used to developing modern web UIs. Also quite limiting in terms of cross-platform UI components.
gfx (only available in ReaScript, not C++ or Rust extensions): Don't have much experience with this, but I know it's some pretty low-level painting primitives. Some scripters wrote quite impressive abstractions that make working with it easier. Still, feels so limited compared to a modern UI framework.

If I would write another serious REAPER extension which requires a modern UI, I would probably choose Rust for the REAPER stuff and Flutter for the UI and let both communicate via WebSockets and/or UDP (as I did with ReaLearn Companion). I guess it all depends on what you need. If you want something that's easy to embed into a parent window, your choices are quite limited.

I encourage you to do so! Getting used to Rust might take a while but it's a very rewarding time investment in my opinion. If you are okay with garbage collection though, then other languages might be move convenient. I chose Rust because I need to do stuff in real-time threads and I was fed up with C++.

I would prefer not to touch C++, the it seems to be the programming-language equivalent of a slow-burning dumpster fire haha. Rust is clean at least, and has sane dependency management + build tools.

If I would write another serious REAPER extension which requires a modern UI, I would probably choose Rust for the REAPER stuff and Flutter for the UI and let both communicate via WebSockets and/or UDP (as I did with ReaLearn Companion). I guess it all depends on what you need. If you want something that's easy to embed into a parent window, your choices are quite limited.

Ah, got it. So if it's to be dock-able, it's SWELL or the highway eh? Guess that doesn't leave much room to bikeshed about options then haha.

And yeah, I think if docking doesn't matter, either Electron + Vue/React (or one of the lighter-weight alternatives) or Flutter with an API-driven UI are definitely the best options.

Thanks, really appreciate your taking the time to respond, will continue to poke around the repo and see what more I can glean =)

helgoboss / reaper-rs

[QUESTION] General "reactive" limitations with REAPER extensions? #30