Explain interaction with contraints

[markafoltz]

The proposal should be integrated with how MediaStreamTrack constraints work.

The explainer does not propose new constraints, nor changes in behavior to constraints. If there is no interaction, then it could say so explicitly.

[guidou]

As proposed in the explainer there isn't any fundamental contradiction with the https://w3c.github.io/mediacapture-main/getusermedia.html#def-constraint-backgroundBlur

The backgroundBlur attribute is functionally equivalent to the backgroundBlur setting. The only difference is that the constraint-based one is boolean instead of a 2-value enum, and is in MediaTrackSettings instead of directly in the track (but settings are an attribute of the track anyway). The constraint model has its benefits:

• It's already there and consistent with other track properties.
• It's allows the application to signal if it's interested in effects, which the browser can use to customize permission prompts.

There are no shipping implementations of the constraints model yet, so it is still possible to update the constraints-based API to incorporate some of the concepts in this API proposal with low compatibility and interoperability risk.

The most significant addition I see in this proposal is the new event to notify about changes to the state of the effect. I think this is valuable and we should incorporate it. It might be worth discussing if it makes sense to have a more general API for settings changes, but I don't think it would significantly change the proposed model.

The attribute for VideoFrameMetadata is largely orthogonal to the MediaStreamTrack API, so no major issues there.

[riju]

As proposed in the explainer there isn't any fundamental contradiction with the https://w3c.github.io/mediacapture-main/getusermedia.html#def-constraint-backgroundBlur

Agree.

The backgroundBlur attribute is functionally equivalent to the backgroundBlur setting. The only difference is that the constraint-based one is boolean instead of a 2-value enum, and is in MediaTrackSettings instead of directly in the track (but settings are an attribute of the track anyway). The constraint model has its benefits:

Yes. We could think of setting as the EffectState , example -

const [track] = stream.getVideoTracks();
const settings = track.getSettings();
const constraints = {
advanced: [{ backgroundBlur: !settings.backgroundBlur }],
};
track.applyConstraints(constraints)
.then(() => {
const settings = track.getSettings();
log(`Background blur is now ${settings.backgroundBlur ? "ON" : "OFF"}`);
})
.catch((error) => log("Argh!", `${error}`));
}

Our initial proposal was to have an enum, but since the platform support on Mac was boolean, we agreed to make it boolean, infact as part of Origin Trial we requested developers if they are happy with boolean or want an enum.

• It's already there and consistent with other track properties.
• It's allows the application to signal if it's interested in effects, which the browser can use to customize permission prompts.

There are no shipping implementations of the constraints model yet, so it is still possible to update the constraints-based API to incorporate some of the concepts in this API proposal with low compatibility and interoperability risk.

The most significant addition I see in this proposal is the new event to notify about changes to the state of the effect. I think this is valuable and we should incorporate it. It might be worth discussing if it makes sense to have a more general API for settings changes, but I don't think it would significantly change the proposed model.

Is the onconfigurationChange EventHandler for example not functionally equivalent ?

The attribute for VideoFrameMetadata is largely orthogonal to the MediaStreamTrack API, so no major issues there.

[markafoltz]

Thanks for the feedback, responding to each comment in order; first from @guidou

As proposed in the explainer there isn't any fundamental contradiction with the https://w3c.github.io/mediacapture-main/getusermedia.html#def-constraint-backgroundBlur

Agreed.

The backgroundBlur attribute is functionally equivalent to the backgroundBlur setting. The only difference is that the constraint-based one is boolean instead of a 2-value enum, and is in MediaTrackSettings instead of directly in the track (but settings are an attribute of the track anyway).

This is probably right. However the behavior of the backgroundBlur setting is not actually defined by the MediaCapture specification that I can find. The setting definition links back to the constraint definition. I do not see steps to set the value of this property or how it gets updated in response to outside events (OS settings, user settings, etc.) Maybe you can help me understand this?

The constraint model has its benefits: It's already there and consistent with other track properties.

True, and now noted in the explainer.

It allows the application to signal if it's interested in effects, which the browser can use to customize permission prompts.

Are you thinking that adding a backgroundBlur: true constraint to getUserMedia() would ask the user whether to enable blur on the camera if blur was available and disabled, and the site did not yet have camera permissions?

There are no shipping implementations of the constraints model yet, so it is still possible to update the constraints-based API to incorporate some of the concepts in this API proposal with low compatibility and interoperability risk.

The IDL has landed in WebKit trunk (tracking bug) but I don't know how close it is to shipping.

I agree the risk is low, based on current information.

The most significant addition I see in this proposal is the new event to notify about changes to the state of the effect. I think this is valuable and we should incorporate it. It might be worth discussing if it makes sense to have a more general API for settings changes, but I don't think it would significantly change the proposed model.

The two advantages for MediaStreamTrack are the event you describe, and future extensibility, which I go into more detail in the updated explainer.

The attribute for VideoFrameMetadata is largely orthogonal to the MediaStreamTrack API, so no major issues there.

Thanks 😸

[markafoltz]

Now responding to @riju

We could think of setting as the EffectState , example

The proposal we have doesn't support setting the state. In your example, how does the application know whether the new setting could possibly succeed? Maybe that is what the MediaTrackCapabilities.backgroundBlur property is for?

Is the onconfigurationChange EventHandler for example not functionally equivalent ?

I took a look at this - what triggers firing this event? How does the developer know what has changed - does it have to keep track of the previous settings / capabilities states and figure it out by manually diffing?

Our initial proposal was to have an enum, but since the platform support on Mac was boolean, we agreed to make it boolean, infact as part of Origin Trial we requested developers if they are happy with boolean or want an enum.

Good to know - did you get any feedback about whether they preferred it? Also I believe Windows uses enums/ constants for effect states and has multi-state effects in some cases.

[markafoltz]

Constraints pose some challenges for dynamic settings of the camera like effects.

The application may be asking for cameras that currently have blur on/off, but this can change, even while the browser is enumerating cameras or offering a permission dialog that allows the user to select cameras.

The application may be asking for cameras that are capable of blur. But blur is implemented by the OS or browser (at least, these are the cases where the browser can detect). So, in most cases this is all or nothing for all cameras, thus not a very useful constraint.

The application may be trying to use constraints to force blur on/off. This has some issues:

1. Currently effects are applied to all streams from the same camera, and thus may impact what is seen by other Web applications currently using the camera simultaneously.
2. This may change a per-camera global state for the blur setting, and thus impact the default the next time the camera is opened by another site, which may or may not be what the user intended.
3. There's no permission model for changing blur on an active stream through constraints.

Were any of these issues discussed during the development of the current constraints model for effects?

How do implementations handle competing requests for other "constrainable" device settings like exposure, pan/tilt/zoom, etc.?

[markafoltz]

We've updated the explainer to not propose any changes to constraints behaviors. The existing constraints will continue to work as they are currently specified, although I believe the spec language should be improved to address some of the issues discussed here.