Describe more precisely transfered track lifetime management.

[youennf]

Fixes https://github.com/w3c/mediacapture-extensions/issues/29.

---

Preview | Diff

[youennf]

This PR needs more work, in particular to add more links to external concepts. Before finalising the PR, it would be good to get input on the overall direction, in particular the lifetime properties.

[alvestrand]

Yes, this needs more work.

In particular, the PR does not consider sources that might themselves be transferable, or might not be relevant to the originating context. The rules suggested (such as privacy indicators) are appropriate for cameras and microphones, and something like this is probably the right behavior for those devices. But generalizing this to all track sources is not appropriate.

I do not believe the addition of a concrete TransformStream as the method for communicating between the source and the track is appropriate at all; it's possible that one can describe this as an abstract operation with the same properties as if one had a TransformStream, but the concrete-sounding description of this non-observable behavior is not useful.

[youennf]

the PR does not consider sources that might themselves be transferable, or might not be relevant to the originating context.

Can you give more details about transferable sources and why we should have special treatment for them? Is it an existing issue or is it to be future proof?

All current 'sources' I know of are tied to their originating context: getUserMedia, getDisplayMedia, WebAudio, RTCPeerConnection, HTMLMediaElement. The PR works for all of them.

Looking at the proposed MediaCaptureTrackGenerator, its writable can be transferred but its writable is not the source. My understanding is that, if the creation context goes away, MediaCaptureTrackGenerator will get ended. The PR also works for it.

I have some difficulties to reason about transferable sources since sources are more a concept than a JS object.

generalizing this to all track sources is not appropriate.

We can extend the algorithm to 'transferable' sources once we have a common understanding of what is a transferable source and whether that is useful to optimize.

For instance, sources could have a boolean [isTransferable] slot. If so, we could transfer the source (like we do for the readable in the current algorithm).

I do not see transferable source handling as a blocker to the current proposal, more as a follow-up issue. If you think that is a blocker, can you detail your reasoning?

the concrete-sounding description of this non-observable behavior is not useful.

An algorithm is usually better as it is more precise. We have good examples in WebRTC specs where we moved from language to precise algorithms, as a way to improve interoperability. We can probably try to use words instead, but what is the advantage?

To help the reader, I am fine adding some text that is describing what our intent is. This PR has some language like that and we could beef it up.

[youennf]

To focus the discussion on lifetime management properties first, I removed the algorithm and made the rules become MUST statements. Feedback most welcome!

[alvestrand]

The two cases I can think of that matter are OffscreenCanvas (if it supports capturing) and GetDisplayMedia; in both cases, it's not clear to me that it's logical to make the track dependent on the original context.

[alvestrand]

.... and now that I think of it, indicators are a browser element, not a renderer element ... which means that sticking with the original frame as the owner of a source flies somewhat against the idea of "transferring an object as a capability" that @annevk argued for.

[annevk]

If this closes #29 a follow-up should be filed as both serialization and transferring are to be defined as two sets of steps.

[youennf]

The spec contains the two set of steps but in a minimal way that does not address lifetime management. An initial version of this PR included an algorithm to improve this set of steps to cover lifetime management. But I removed it for now to concentrate on the properties we want, before trying to provide an algorithm to implement these properties.

[alvestrand]

We should probably put the horse before the cart and concentrate first on what lifetime properties we want, and document those.

[jan-ivar]

This is editorial. MediaStreamTracks already have well-established lifetime properties.

Sure, making them transferable challenges us to revisit those properties, but until we do, they are still there — because a lot of privacy and security considerations rely on them — so we should clarify them vis-a-vis transfer until we decide to change them.

Any concerns over decision lock-in seem unwarranted, because these assumptions are already there. We just made a mistake introducing ambiguity that wasn't there before. But such new ambiguity does not override previous decisions.

Thus this PR is the conservative way to iterate here and should be merged asap IMHO.

[youennf]

I honestly thought we had consensus on these properties when we discussed them during March interim and at the editor's meeting we decided to merge the initial PR specifying the two sets of steps. This PR is trying to reflect that in English and not as an algorithm based on your feedback.

getDisplayMedia tracks, like getUserMedia tracks, are bound to the context they are created in. I validated that Chrome, Firefox and Safari all stop getDisplayMedia tracks when its context is navigating away. We are building on that to be as conservative as possible. This does not prevent to be more ambitious in the future if we have further use cases.

As of OffscreenCanvas, we cannot capture from it but from a canvas dom element which is not transferable.

[alvestrand]

I will have to return to this issue two weeks from now, but sketching out some approaches (that need detailing).

One approach:

In the model of [[mediacapture-main]], a track is inextricably linked to a source. Sources are non-JS-visible objects that exist in a single Javascript context, and it is part of the definition of a source that it can never be moved from that context, and dies when that context dies.

A transferred MediaStreamTrack retains its linkage to the original source; when the context dies (this term has to find its exact definition), the source ceases to exist, and the track ends. This is a cross-context observable event.

[alvestrand]

Another approach:

In the model of [[mediacapture-main]], a track is inextricably linked to a source. Some sources are not JS-visible objects; others are JS-visible objects, which may or may not be transferable.

The currently defined sources of tracks include:

• Devices. The process of getting access to a device is considered to create an object that is the "source" for the track coupled to that device, but also encompasses its privacy indicators and liveness indicators. This non-JS-accessible object is not transferable, and disappears when the context ends.

• DOM objects. The DOM object is considered the source itself, and the track remains alive as long as the DOM object remains alive. The track is considered to hold a strong reference to the DOM object, so garbage collection will not cause the DOM object to disappear, but the destruction of the DOM object's context will cause the source to disappear and the track to end.

• Other objects. The definition of new objects as track sources MUST define how the lifetime of the resulting track is managed.

[youennf]

This seems like a good addition to me. I filed https://github.com/w3c/mediacapture-main/issues/804 since I think part of this text should go in mediacapture-main. Only the bits specific to transferability should be added here.

I would tend to go with proposal 1. In my mental model, a DOM object or a device is not a source. A source is created based on either a DOM object or a device and is tied to its context. Also, all JS-visible objects that happen to be a MediaStreamTrack 'source' are not transferable so the distinction between devices and JS objects does not bring much.

[youennf]

I uploaded a PR in mediacapture-main to tackle this issue at https://github.com/w3c/mediacapture-main/pull/805. In mediacapture-extensions, we could update the transfer algorithm so that transferedTrack.shouldStopSource is set to false. In that case, the source will be stopped when all its related tracks are stopped (or the source original context goes away).