User Defined Data Streams for Direct Hardware Communication

[lxw404]

Is your feature request related to a problem? Please describe.

Currently, hardware integration is limited by the pace of platform development, however Resonite has the potential to be a powerful testbed for hardware development directly.

It would be extremely beneficial from a product and end-user standpoint to allow users to specify their own ValueStream sources which can be linked directly to hardware through drivers that they write themselves rather than relying on clunkier WebSockets implementations, or requiring a feature request for every single hardware device that exists (or does not yet exist).

Describe the solution you'd like

An effective solution should include a way of defining new ValueStream sources as well as a type of output mechanism (OutputStream or something) or even some sort of generic stream driver. These would then map values to a region of shared memory to allow inter-process communication between Resonite and whatever drivers are written.

The user should be able to:

• Add and configure as many devices as they need (config file, dynamically, whichever way this would make sense)
• Map device inputs to in-game streams with a specific data type
• Read values from input streams into in-game systems
• Map arbitrary in-game values to output streams
• Drive/Write values to output streams

Additionally, it would be useful to specify if a stream is purely a local value (in the case of some devices, the values only affect the end user in a specific way and don't necessarily need to be networked).

Describe alternatives you've considered

WebSockets is the only supported alternative at the moment that is generic enough to handle this, but suffers in many ways:

• It lacks performance for devices which require near instant feedback (prototype haptics devices, sensors, robotics, etc.)
• The implementations become clunky due to the way it requires data to be sent and interpreted (Resonite's WebSockets implementation currently only supports strings, an operation which itself already costs a lot in terms of parsing, packing, unpacking, and re-parsing, whereas raw data usually can just be interpreted as direct binary values, and even with raw data sending, a network stack is yet another unncecessary performance hit)
• Users need to manage and run their own server(s) for sending/receiving data

Native integration is also possible, but requires manual work on the part of the team to do so and is unrealistic to expect every single variant of every hardware device to be implemented.

Plugins are an alternative, but it forces version incompatibility.

Mods have achieved this partially in the past in a different context (FFT Mod)

Additional Context

No response

[hazre]

I also think this would be very beneficial. I've already created a mod that uses ValueStreams to add support for a Eye and Face tracking interface such as VRCFaceTracking. I create the value streams for all individual blendshape parameters and then read them in-game using dynamic variables by creating a DynamicValueVariable for each stream in new slot under avatar root.

Theres definitely room for improvement but it works and It's modular enough for any One-way IPC.

[BlueCyro]

+1 for this, but also almost certainly requires collections, or at least would need them to be at all useful.

[Geenz]

We intend to support user provided hardware drivers moving forward after the data model rework that won’t break compatibility between clients. We don’t presently have an issue for this- however we’d do that before taking this on.

[Frooxius]

Our plan is to add ability easily implement support for new types of hardware. One of those ways would be to support loading device "drivers" - DLL's that provide support for particular hardware and feed it into the input systems.

I'm not sure if we'd do things specifically the way it's requested here. Off the cuff, one thing that concerns me a bit is that it seems to couple things too tightly together, whereas our general approach is to build things from independent systems.

E.g. you can already make arbitrary ValueStream systems from code - there is nothing about them that would make them work only with specific stuff we provide.

What would help there is a way to easily create and link them to parts of data model - that way you can say "This value gets synced via streams". It would be independent of use - you could use to stream hardware data, but also anything else too. They wouldn't care about any sort of IPC communication - it's not their responsibility to.

The other issue that I see from this is way to feed in the data from the hardware. I'm not sure if there's a good unified generic solution. IPC is a mechanism, but it doesn't provide any particular protocol - and you'd probably need to run a server still too to push it through the mechanism and whatever structure it needs.

I think having device driver DLL's would work best in this case - you just write code that interfaces with the hardware in the best possible way, rather than trying to push it through some standard system - otherwise you don't get around the need to run other software/server to feed the data in and format it.

[shiftyscales]

Closing this issue. As Frooxius had determined above, the proposed approach probably wouldn't be implemented as it is presented here. Please focus on the underlying issue when opening a feature request, rather than a specific/highly detailed way to implement it.