Open jvdrhoof opened 1 month ago
The place where the high level decision is communicated to the lower layers is PointCloudPipelineOther::SelectTileQualities()
.
At the moment the implementation of this is a bit of a mess: for each of the transport protocols that implemented multi-quality we try to cast this.reader
to the type, and if it works we call the method needed on that casted object.
The quickest solution is to simply add another cast and if statement for the WebRTC reader.
Then later we can look at the pattern and add a general method to the ITransportProtocolReader_tiled
interface.
Note that this will give the WebRTC reader the decision for one client_id
only, but I think @jvdrhoof said that was good enough for now. Especially for the mmsys measurements where we're initially only interested in the 2-user case anyway.
Oh yes: note that the tile and quality indexes are into the IncomingTileDescription[]
and its IncomingStreamDescription[]
that you got passed as a parameter during the Init()
call.
And I think that the streamIndex
inside that last structure is a Dash implementation detail, but it may be reusable for WebRTC. Ultimately all the information here is gotten from the PointCloudNetworkTileDescription
which is created on the sender side, and then communicated to all receivers through the VR2Gather session communication.
The point cloud pipelines will delay creating the receivers until this information has been received, so in the transport protocol Init()
method you already know everything about all streams/tiles/qualities that will be available for this sender.
I extended RegisterTransmitter
so that the number of tiles and quality representations can be passed as input argument to the WebRTC client. When additional encoders are added in config.json
, the number of available quality representations is now correctly registered.
I also managed to retrieve the index and quality of a particular tile/stream in WebRTCStreamDescription
, so that this information can be propagated. While maybe not the cleanest approach, the AsyncWebRTCWriter
now has all information needed to send tiles out at a specific quality, fully compatible with our WebRTC DLL/Go code.
I further managed to include an AsyncWebRTCReader_Tiled
in PointCloudPipelineOther.cs
, which contains setTileQualities
with decisions for every tile passed as a list. This function forwards the decisions through setTileQualities
in TransportProtocolWebRTC
. The latter generates a control message to the SFU to update the quality decisions.
What I do notice, is that dynamic quality decision-making is not triggered by default. To this end, BaseTileSelector
should be run inside a thread, with the Update
function calling StartCoroutine(_doSelectTileQualities(selectedTileQualities));
where needed. What configurations do I have to modify to get this to work for WebRTC?
Provide quality decision-making for the existing adaptive bitrate heuristic used for LL-DASH. Decisions on the quality of the tiles should happen in the following format:
E.g.,
0,1,1;2,1,−1,1,0;3,1
sent by client 1 indicates that:Messages can be sent using the
send_control_packet
function, made available through the WebRTCConnector DLL. It is not obligatory to send decisions on multiple clients or point cloud objects per time.