search

w3c / mediacapture-extensions

Extensions to Media Capture and Streams by the WebRTC Working Group
https://w3c.github.io/mediacapture-extensions/
Other
19 stars 15 forks source link

Transferable MediaStreamTrack: Within Agent Cluster or between Agent Clusters? #43

Open alvestrand opened 2 years ago

alvestrand commented 2 years ago

https://w3c.github.io/mediacapture-extensions/#transferable-mediastreamtrack does not specify any restrictions on transfer of a MediaStreamTrack.

When investigating the possibility of implementing this extension, we found that there were fundamental differences in the complexity of transferring within an Agent Cluster (basically same-origin iframes and DedicatedWorker) and transferring outside an Agent Cluster.

It's fairly obvious that there are potential use cases for cross-cluster transfer, but it's also obvious that supporting them is costly.

Should we restrict transfer of MediaStreamTracks to within an Agent Cluster, or should we require universal transferability?

alvestrand commented 2 years ago

ping @youennf @jan-ivar

youennf commented 2 years ago

We should first decide the long term goal. In general, it seems we want universal transferability and user agents already have the infrastructure to efficiently transfer audio and video cross process. Also, I am not sure we have examples of partially transferable objects (OffscreenCanvas maybe?).

Practically speaking, it seems fair to start with the easiest (and probably highest priority) scope of transferring to DedicatedWorker. It would be good to be able to feature detect this transitory situation, maybe by having implementations only exposing MediaStreamTrack to DedicatedWorker at first.

eladalon1983 commented 2 years ago

I think complexity can also be different depending on the "actual type" (explained shortly) of the MediaStreamTrack. By which I mean, a video track derived from a call to getDisplayMedia can be very different from one derived from a call to HTMLCanvasElement.captureStream, and that in turn different from a track derive from camera-capture. I wonder whether subclassing MediaStreamTrack (as I had previously suggested) could help with track transferability.

In the inheritance tree sketched below, the first three immediate descendants of MediaStreamTrack are CanvasCaptureMediaStreamTrack, DisplayCaptureMediaStreamTrack and PeripheralMediaStreamTrack. Transferring at least DisplayCaptureMediaStreamTrack sounds like it should be easy-ish between Agent Clusters. And subclassing means that we could define different transferability levels to different types of tracks. So maybe Canvas-tracks can be transferred in the same Agent Cluster, but gDM-tracks can be transferred anywhere.

Screen Shot 2021-11-30 at 16 06 02

-- Note: The "suggested" part of the inheritance tree above is because this slide was lifted from my presentation to the WG about Conditional Focus. Please disregard that part.

youennf commented 2 years ago

It is not clear from the spec whether transferring a MediaStreamTrack preserves the subtype. For instance, are we expecting transferring CanvasCaptureMediaStreamTrack to end up being a CanvasCaptureMediaStreamTrack? Probably, but we would probably need to state something like this in CanvasCaptureMediaStreamTrack spec.

The same fuzziness in specs happen for cloning a CanvasCaptureMediaStreamTrack track. It might be worth doing the necessary clarification effort for CanvasCaptureMediaStreamTrack before introducing further MediaStreamTrack children.

alvestrand commented 2 years ago

Should we open a new issue for the question of whether subtype is preserved?

youennf commented 2 years ago

I filed https://github.com/w3c/mediacapture-extensions/issues/51 for transfer + subtype. And https://github.com/w3c/mediacapture-main/issues/857 for cloning + subtype.

youennf commented 2 years ago

For CanvasCapture track, we cannot transfer the canvas element, so the transfer step would anyway be lossy. We could still make requestFrame() work but it does not make a lot of sense in any scenario that I can think of.

alvestrand commented 7 months ago

The initial reason for my question was because I think that transferring things between agent clusters (that definitely open in a different-process, nothing-shared model) may have a very different complexity from transferring within an agent cluster (where there may be many things shared in the implementation).

Another reason is that transferring MediaStreamTracks inside an agent cluster seems to be relatively easy to reason about from a security standpoint; transferring MediaStreamTracks outside of an agent cluster (into a different origin) may have very different security implications.

youennf commented 7 months ago

transferring MediaStreamTracks outside of an agent cluster (into a different origin) may have very different security implications.

In terms of security, there should be strictly no difference. The privacy indicators stay where they are, the track is transferred but the source stays where it is.

The model is that the source of the track is tied to the mediaDevices that created it and is added to https://w3c.github.io/mediacapture-main/#dfn-mediastreamtracksources. When the context goes away, the source is stopped.

We try to be clear about this in the main and extension spec, I am not sure what additional clarification we can bring.

That said, it seems we are missing the MediaStreamTrack creation steps in the screen share spec, we should add this for correctness, even though https://w3c.github.io/mediacapture-extensions/#transferable-mediastreamtrack outlines what needs to be done.

youennf commented 7 months ago

I filed https://github.com/w3c/mediacapture-screen-share/issues/293

alvestrand commented 7 months ago

The additional security consideration is that the media info is now available in a context that the user did not authorize. We know that it is possible to get almost the same level of access by setting up a PeerConnection to transfer the media, but any time new data is shared between different origins, people make security noises.