MediaStreamTrack resizeMode constraint is too vague to request specific strategy

[anthmFS]

Currently, when you want to ask for a specific resolution using height, width, aspectRatio and resizeMode there is no way to specify that you do not want to crop.

Situation: You want to use 640x360 res. The camera supports 640x480, 1280x720, and 1920x1080 Despite the resizeMode "none" or "crop-and-scale" Chrome for instance will take the 4:3 640x480 and crop it to 16:9 ratio which effectively zooms the image leaving barely enough room for someone's head to fit.

There is not a way using the spec to insist that you want to scale down the nearest 16:9 res (1280x720) to 640x360 Even if you say you want an exact aspect of 16:9 it still chooses the crop technique.

So the spec needs to be precise enough to entice the browser developers to implement something that allows this specificity.

Perhaps a resize mode of "scale-only" or "preserve-aspect-ratio" or just some interpretation of a combination of existing params such as "crop-and-scale" when used with an exact aspect ratio, limits the source video to those who meet the required aspect ratio.

[anthmFS]

also a "letterbox" option so you can choose to downscale shared applications locally via screen share.

[jan-ivar]

Despite the resizeMode "none" or "crop-and-scale" Chrome for instance will take the 4:3 640x480 and crop it to 16:9 ratio which effectively zooms the image leaving barely enough room for someone's head to fit.

I'd say this is a bug in Chrome. none says: "This resolution is offered by the camera, its driver, or the OS."

There is not a way using the spec to insist that you want to scale down the nearest 16:9 res (1280x720) to 640x360 Even if you say you want an exact aspect of 16:9 it still chooses the crop technique.

Good point. I think I like the idea of standardizing a "scale-only" mode better than user agents trying to infer a client's preference from whether they used exact or not with aspectRatio.

Note, if this is for web conferencing, you may be able to work around this problem by using scaleResolutionDownBy.

[jan-ivar]

Options:

1. Work around it using track.clone and set one to a high-res 16:9 mode using resizeMode: none and the other to the resolution you want. This should force the implementation to open the higher res (in theory, may not work on all browsers yet)
2. File a bug on browsers to stop cropping 4:3 to 16:9 (though there's a performance cost I hear)
3. Since user agents are ultimately allowed to choose which mode to pick, ask them to look at aspectRatio as a hint (might have poor web compat)
4. Define a new scale-only value in the spec specifically to disallow cropping.

[henbos]

@guidou What is a sensible way forward here?

[guidou]

I think the best way to achieve what @anthmFS wants is to define a "scale-only" resize mode. It shouldn't be very difficult to implement if you already implement "crop-and-scale", since it's basically a "crop-and-scale" where you maintain the original aspect ratio (so that you don't have to crop).

[alvestrand]

Given a source of 640x480 (only):

If constraints are exact 640x360, then "scale-only" doesn't make any sense. If constraints are ideal 640x360, then "crop-and-scale" will give you 640x360 cropped; "none" will give you 640x480. If constraints are ideal w=640, then "crop-and-scale" shouldn't give you a reason to crop; aspect ratio can be preserved.

The middle case is the only one where "scale-only" makes a difference, I think.

[guidou]

Consider @anthmFS's case. He wants 640x360 res, but the camera supports 640x480, 1280x720, and 1920x1080 and uses "crop-and-scale".

He wants the camera to be opened in 1280x720 with the track producing 640x360 using only scaling. However Chromium opens it in 640x480 and the track produces 640x360 using only cropping.

Based on the spec, both strategies are perfectly valid. Chromium chooses the latter because it always tries to use the closest (and thus smallest) resolution (in size) that can be cropped-scaled to the ideal resolution. This is 100% spec compliant, however it is not what the user in this case would prefer, and I think preferring only scaling is a legitimate thing to want and there is no way to tell the browser to apply just scaling to the source frames.

[anthmFS]

I think the "scale-only" makes sense and solves the problem. Especially since the number one thing shown in video from these cameras is probably a person and vertical crop is only going to make it zoom more. When turning native 16:9 into 4:3 then horizontal crop makes sense.

There are similar uncontrollable behaviors in params for screen sharing which might be solved by the same change. When I share my 4k screen when I know it will eventually be displayed on a 720p canvas, I may ask the browser for 720p in advance to spare the intense bandwidth of sending the native 4k over the peer connection which I know will be eventually downscaled anyway. This used to work and the share was letterboxed into the requested res was sent but at some point, the same interpretation of the spec results in scale and cropped screen instead of letterbox despite what combo of params are given.

[alvestrand]

Re-reading the thread, I think I agree that "scale-only" makes sense. The issue here is that we have a camera capable of 2 modes, one close to the desired size, and the other having the desired aspect ratio.

of course, specifying "scale-only" on a browser that does not support it will be ignored (if ideal) or cause failure (if exact).

[anthmFS]

Thanks, what would the next steps be?

[anthmFS]

Bug opened in Chrome https://bugs.chromium.org/p/chromium/issues/detail?id=1380442

[anthmFS]

Any particular steps to make progress on this?

[anthmFS]

@guidou, What are good next steps? The folks in chromium are waiting for the spec to be clarified before they engage. I opened this bug 3 years ago and everyone seems to think I have a reasonable point. I am a n00b at getting specs passed.

[alvestrand]

1) Get someone to write a pull request on the spec adding "scale-only" mode with a description of what it does. 2) Raise a Chromium bug asking them to implement what's (now) in the spec.

Because of our desire to get mediacapture-main pushed to REC, we are unlikely to add new features there at the moment, which is why this thread is on the mediacapture-extensions repo.

[anthmFS]

How about now, a year later? Who would that someone be btw?