Wolf API

[ABeltramo]

Overview

The basic idea is to properly separate the core of Wolf from the Moonlight protocol:

The main benefit of this approach is that the same underlying core methods (create virtual remote sessions and stream them) could be accessed by any other kind of client, for example:

• A web config page where you can see live what's running and have full control over start/stop of any part of it
• A different streaming protocol like WebRTC
• As a normal API accessible from any other kind of language

This would also allow controlling multiple distributed servers (aka: Kubernetes support, as requested by many)

Implementation

So... a REST API?
A typical REST API could work, but it doesn't cover all the events that are starting from the backend and that needs to be sent to clients, ex: an application stopped running, a pipeline failed, or a remote controller received rumble data.

So... a WebSocket?
A WebSocket allows having a communication channel that is bidirectional, and has the good advantage that can also be used natively by browser based languages. This might be one way forward, but then I discovered something that might be a better fit:

MQTT:

• Event driven, a great fit to how Wolf is already structured
• Can work over TCP/UDP and WebSockets.
• Supports QoS, persistent sessions, encryption and authentication
• Lots of well-supported libraries to use it in most programming languages

My only concern is how well it'll be suited for carrying mouse/keyboard/joypad inputs where latency is absolutely fundamental.

This still needs to be fleshed out properly, in the meantime, here are a few interesting links:

• Broker: eclipse/mosquitto or emqx/nanomq
  • Latency, see: https://github.com/eclipse/mosquitto/issues/1616 and https://github.com/eclipse/mosquitto/issues/433
• Using AsyncAPI it's possible to formally define the API and even automatically generate the boilerplate code for clients.
• MQTT Debug app

[a-priestley]

Hi @ABeltramo, Have you considered gRPC?

[ABeltramo]

I have looked briefly into it, I think it wasn't well suited for browser based apps or it was lacking in some other department. I should double check though!

[a-priestley]

I work with web-based grpc technology professionally. It is possible to achieve server-side streaming with it, but duplex is not yet possible -- from the client, only unary calls are supported. However I know that you can fall back to websocket streaming from the client, so I wonder if the same functionality can be achieved with MQTT instead of ws? Though this does raise the question of whether you would need a true duplex connection for your API, because as far as I'm aware, the fallback protocol would essentially create two connections: one server-side (gRPC), and one client-side (ws or hopefully MQTT)

[ABeltramo]

Yeah we absolutely need duplex since some stuff will come from the client (like joypad inputs) and some stuff will come from the server (like joypad rumble) and the link must be as low latency as possible for obvious reasons.

My idea in the end was to start with a simple plain websocket and see how that goes. It should be trivial to swap one technology with another one once everything is properly setup in code.

[a-priestley]

Keeping to a more mature technology to start out sounds like a solid plan! I'm curious though if you are planning on opening up a full duplex stream through which to transmit every type of data? The reason I bring this up is because I'm thinking about things like video and audio versus things like input. Video and audio could conceivably be solely server-sent, where input would be only client-sent no? Would either of these transports need to be two-way as opposed to a one-way stream for each of them?

[ABeltramo]

Audio and video will surely go over WebRTC, but we need an additional bidirectional stream for user inputs and general server events (app start/stop for example).

input would be only client-sent

This is something I also wrongly assumed before working on this project but it's not as simple as that, rumble (and more recently force feedback for PS controllers) are events that originates from the remote game running and needs to be sent all the way to the local gamepad from the server.

There are other "minor" things like setting the gamepad led color for example which is something that originates from the app remotely and not from the local gamepad.

[tutman96]

I’m working with Farcry to get a setup using sway and webrtc to do video+audio and gamepad. We’re going to be using WebRTC DataChannels for input. DataChannels are pretty neat in that you can specify packet ordering, retransmits etc (https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/createDataChannel). We’ll be using MQTT for signaling and for things like periodic server/client stats being broadcast around.

[ABeltramo]

Thanks for pointing out DataChannels I didn't knew about them, but it sounds like a very nice fit for what we need!

[SamuelDudley]

If you looking to the bleeding edge you might consider WebTransport over HTTP/3 (QUIC) for a data transport layer.

Additional ref: https://github.com/kixelated/moq-rs

As QUIC is over UDP it opens a lot of interesting options you dont get with WebSockets, MQTT and the like.

[ABeltramo]

Since this was opened I've further separated the modules into independent projects, like inputtino and gst-wayland-display. I don't exclude that we might re-visit exposing an API for Wolf in future, but the focus will be different compared to what it's outlined in here..