search

jackaudio / jack2

jack2 codebase
GNU General Public License v2.0
2.2k stars 373 forks source link

Open Sound Control support #513

Open piegamesde opened 5 years ago

piegamesde commented 5 years ago

There already is https://github.com/ventosus/jack_osc, but I consider this more a hack than anything else. I'm playing with the thought on implementing proper OSC support in Jack as a new port type. Can anyone with more knowledge of the codebase please hint me where to start and what to change?

falkTX commented 5 years ago

there is quite a lot of it involved. not just defining the macro for the port type, but also a header file for the struct type and functions to read and write osc. then the engine code to pass that data around and mixdown when multiple connections hit the same target port.

piegamesde commented 5 years ago

Shouldn't the mixdown work the same as with MIDI? Simply transport the content as binary data (with timestamps) and let the clients handle the interpretation

falkTX commented 5 years ago

yes it works the same as midi. messages are mixed down in order of their timestamp. copying and modifying midi code is the best way to get osc

piegamesde commented 5 years ago

@falkTX Okay, do you have some files for me that I should look at first?

piegamesde commented 5 years ago

Also, should I really duplicate all the MIDI code or should I rather try to add an abstraction, so that 1) we don't have to maintain dupliate code and 2) other transport protocols may be added later on?

beatboxchad commented 5 years ago

I've been daydreaming about this lately, too, and I just wanted to note that I think the solution in jack_osc is less of a hack and more of a really elegant insight and a complete solution. But I agree it's more of a proof of concept, so it's fun to think about how to flesh it out.

This might be a case of premature optimization. Do you have a use case for routing OSC between local applications in this manner? To my understanding, you'd need to implement client code in each app you want to use it with, to listen for OSC messages on the JACK bus. Whereas a loopback interface is already available to route these messages to any OSC-controllable application in a generic cross-platform manner. Would this confer any meaningful performance benefit?

My initial guess is no. I think UDP over the loopback is unlikely to introduce any detectable latency, and if it did, using JACK instead would be unlikely to fix it. The kernel driver that runs the loopback interface is basically doing the same thing with UDP packets as JACK does with MIDI.

So we would have all this extra work coding audio applications to support listening for OSC messages on JACK with no benefit.

All that said:

I don't have the code in front of me, so please excuse anything I am missing, but it seems to me that incorporating it into JACK2 involves little more than refactoring. Rename the common internals something more generic that reflects their purpose; use them in the OSC and MIDI headers or whatever (my C++ terminology is rusty).

Cheers!

piegamesde commented 5 years ago

First off, to the reason why I want OSC in Jack: I spent a good afternoon trying to set up a few applications and let them do stuff with music data over OSC. I didn't really get that far, because most of them only use OSC for metadata/control, connecting things with UDP ports is a pain and there is little to no documentation. My idea was, that if OSC became a first class citizen of JACK, including patchbays, session management and the same interface as midi, then audio programmers were more likely to implement proper OSC support as MIDI successor/alternative. So I'm not thinking about performance, but about usablity, a unified API and software adoption.

The thing I dislike about jack_osc is that there is no semantic separation of data, e.g. in the patchbays.

I played around a bit with the code and have the following implementation proposals (disclaimer: I only looked at the headers yet, not at the actual implementation):

1.

2.

3.

The options essentially boil down to "have a separate port type for each protocol" and "have a separate port type for different data handling + use metadata". There are many more variants and combinations of both, but these are the three I see fit best.

beatboxchad commented 5 years ago

I didn't think about that part. Yeah, that's true. You gotta know a little network to get things done with OSC currently, and the patchbay interface is also pretty sweet.

HaHeho commented 5 years ago

First off, to the reason why I want OSC in Jack: I spent a good afternoon trying to set up a few applications and let them do stuff with music data over OSC. I didn't really get that far, because most of them only use OSC for metadata/control, connecting things with UDP ports is a pain and there is little to no documentation. My idea was, that if OSC became a first class citizen of JACK, including patchbays, session management and the same interface as midi, then audio programmers were more likely to implement proper OSC support as MIDI successor/alternative. So I'm not thinking about performance, but about usablity, a unified API and software adoption.

I would say that Jack (including MIDI) and OSC already achieve quite well what they're designed for. Having their independent use cases and different ways of using them, with plenty of overlap of course. Software that has a use for both protocols (in the context of audio/music/sound that is quite many of course) already have them implemented independently. If you as a developer see use for that, you can quite easily implement them with a multitude of APIs in any language as well.

Hence my question: what is the gain of strongly integrating OSC into Jack? Is it substituting MIDI with OSC by forcing it into the same use paradigms of "virtually plugging 1:1 cable connections"? ... I would argue that is not necessarily what it is designed for, however should already be possible to implement if you have the support. Is it receiving "free OSC support" via Jack for applications / plug-ins that do not support it yet? ... That is a good point of course, even though I have no idea in how many cases that actually applies. At the end of the day, your application under consideration has to handle, i.e. implement support, for OSC messages anyhow.

Would "simply" and solely incorporating OSC into Jack actually solve the problems you encountered? Does it open further possibilities? Does it yield a general boost in usability?

beatboxchad commented 5 years ago

These are good questions.

I do find myself intrigued by the idea of not having to understand computer networking at all in order to exchange OSC messages between applications. I got into open source pro audio to lessen the financial and technical barriers for creatives who want to produce.

Indeed, I was not productive with OSC until I learned networking. Implementing a separate transport layer for OSC messages is a Big Idea which totally does trigger a few of my slow-down-and-think-about-this developer heuristics, but once it was framed in terms of that user experience, it clicked for me.

I'd like to help!

falkTX commented 5 years ago

Note that the biggest hurdles with MIDI have been solved in MIDI v2, so that is worth investigating before dedicating a lot of time into jack+osc.

beatboxchad commented 5 years ago

Right now, you route OSC messages with hostnames and ports. You gotta have mDNS configured, you need a local network. It's a pain. Apps are good at auto-configuring but it's a black box to the non-technical. It took me years to get off my butt and start using it, and I had to have a lot of clarity about my production process before it was worth the effort.

The protocol itself is a huge boon, and the network transport is still necessary across hosts, but as I come up for air from writing this big pile of SuperCollider code to route and respond to OSC messages between Ardour, Touch OSC, SC, and so on, I understand why this tech is only accessible to developers as of now.

Whether this problem is JACK's responsibility to solve, and what the ideal solution looks like, are still good questions.

A patchbay sounds great though. Worth noting: we don't necessarily need the new JACK-powered transport layer to write such an interface (ie tight integration with JACK necessary to solve this problem?).

beatboxchad commented 5 years ago

Remember that JACK and QJackctl are two different things. It seems like could might use the existing UDP transport, and just write the front-end you have in mind.

piegamesde commented 5 years ago

Maybe the focus on OSC is a bit too narrow.

JACK audio has support for sending (arbitrary) real-time event data called "MIDI ports". Jack has a good API, and a whole lot of tools that come with it: Patch bays, session management, etc. . But why should we limit ourselves to MIDI, if the same infrastructure can be used for any message protocol with little effort? OSC is just the one example I'd like to use. There could even be MIDI2 support, but we should think about how to handle its bidirectional communication. (I honestly don't believe in MIDI 2 until I see some actual hardware support).

Say I wanted to patch a Jack MIDI application for OSC support. I'd have to:

The last two points are those I want to get rid off through JACK support. Opening up + configuring a server is way more work than changing the port type and the process callback.

Is it substituting MIDI with OSC by forcing it into the same use paradigms of "virtually plugging 1:1 cable connections"?

I think so. I see OSC as a spiritual successor of MIDI: Both are unidirectional, event-driven, binary protocols with a loose emphasis on a default transmission medium. With Jack, we ignore the default transmission medium for practical reasons and extend the connections from 1:1 to 1:n and n:1.

Whether this problem is JACK's responsibility to solve, and what the ideal solution looks like, are still good questions.

This may not be the best solution, but a good solution is worth nothing if nobody uses it. We've already done 99% of the work without really knowing, the only thing that's missing is proper branding and a bit of backwards compatibility. Why should we write OSC discovery protocols and patchbays (I've thought a lot about those and even seen some OSC alternatives/extensions that tackle this) if we can patch our existing tools with ease?

I don't know if it is going to work out the way I hope it will. Maybe it will simply end as one of those obscure niche features nobody really uses. But given the potential this has and how little work remains to be done, I'd like to move the focus of this discussion from if to how to implement this feature.

mathiasbredholt commented 4 years ago

Check the libmapper library for reference, it is a similar platform using multicast to connect devices with OSC

7890 commented 4 years ago

...but I consider this more a hack than anything else.

Look closer, it's a pretty elegant, non-invasive approach that can even scale to other formats.

Add an OSC library to handle packets

Certainly you'll need a library to handle OSC data. However adding that library to JACK is debatable. It would imply to add libraries for any future byte format that might be useful to ship via the original MIDI ports to JACK.

Shipping any data via MIDI ports is already possible and it's up to the developer to link clients with the needed libraries to handle arbitrary data on MIDI ports and also care about anything related to network enabled features such as sending OSC into and out of JACK (from external, non-JACK processes). The things to solve are mainly of semantic nature. Metadata and a set of well-known keys / properties can solve some of this.