Implement a ZeroMQ SFU

[jackjansen]

Update 20-May-2024: this issue has been split into two, with the title of this one changed because most of the information here is about a reflector that isn't as simple as it can be.

We need a very simple SFU that can just forward TCP streams.

Once we have that, and we've implemented using that in VR2Gather, we don't need the SocketIO media transport anymore.

And once that is not needed, we can look at other socketio implementations for VR2Gather.

And that means we can replace besthttp by another, open source, socketio implementation for VR2Gather.

And that will make life a lot easier for the entry to the ACMMM open source competition.

[jackjansen]

First implementation idea. @troeggla comments please.

Everyone connects to the same port on the tcp-sfu. Everyone has a single connection. The client can send a few commands:

PUBLISH 0 streamname\n
SUBSCRIBE 0 streamname\n
UNSUBSCIBE 0 streamname\n
DATA 12345 streamname\n12345 bytes of binary data

The server will simply forward each of the DATA messages on the outgoing connection to everyone who has SUBSCRIBEd to that streamname.

It is an error to send DATA for a streamname that you haven't PUBLISHed. So there's probably one other unsolicited message from server to client

ERROR 0 message\n

[jackjansen]

The streamname needs some sort of structuring. It definitely needs an indicator of the source client (probably the user UUID, which is what we use in various places).

It needs a media type and/or 4CC. Possibly 4CC is enough.

It needs a tile number, and possible a quality level. At the moment, available tile numbers are communicated between VR2Gather instances separately (by the TilingConfigDistributor and its TilingConfigMessage) but this information doesn't include a quality level. Only DASH supports multi-quality at the moment, and it communicates the multiple available qualities in the manifest file. While we're modifying all this stuff we might as well address this.

It may be good enough to simply structure the streamname with slashes, for example

03cbbc3a-59fa-4f75-a68b-466914767506/pointcloud/0x31697763
03cbbc3a-59fa-4f75-a68b-466914767506/pointcloud/0x31697763/1
03cbbc3a-59fa-4f75-a68b-466914767506/pointcloud/0x31697763/1/2

(untiled point clouds, tile 1 of tiled point clouds, quality 2 of tile 1 of tiled point clouds).

For now we can re-use the sfuData.url_gen field to communicate the relevant information from the orchestrator back to the clients, keeping url_audio and url_pcc as they are encoded now. For a client like the above, we would get something like tcp://192.168.1.123:9091/03cbbc3a-59fa-4f75-a68b-466914767506 as the value. The transport protocol handlers in VR2Gather could get all the needed information from that.

[jackjansen]

Hmm. I'm wondering if all of the PUBLISH messages should be forwarded to all of the connected clients (and also remembering all PUBLISH messages ever seen, and forwarding them to new clients upon connection).

That way, the clients could dynamically determine which potentially interesting streams are available (for example to determine the available qualities). But this may require adding a parameters opaque string parameter to PUBLISH.

Hmm. the 4CC could be in that parameter...

[jackjansen]

@jvdrhoof comments please on the last two remarks, how easy it would be to also use this for webrtc (the idea of the single URL to communicate all the needed information about the webrtc-sfu to the clients). This would also be the way we communicated the unique small-integer clientId to the VR2Gather instances.

[troeggla]

Sounds reasonable to me. Maybe it would be worth it to implement this on top of ZeroMQ, which is very lightweight but, has a PUBSUB mechanism built-in and would take care of all the connection management out of the box.

https://zeromq.org/

[jackjansen]

@troeggla nice idea! I was going to do a quick-and-dirty Python implementation, but it looks like zeromq takes care of all the grotty details. I'll have a look.

[troeggla]

I have used it in the past for the EmpaticaRelay app to implement the communication between the smartphone app that reads the data from the Empatica wristband and the data collectors/visualisers. The networking code on either side was not more than a handful lines of code.