search

w3c / mediacapture-extensions

Extensions to Media Capture and Streams by the WebRTC Working Group
https://w3c.github.io/mediacapture-extensions/
Other
19 stars 14 forks source link

Add a blackFrames counter to MediaStreamTrackVideoStats #144

Open guidou opened 1 month ago

guidou commented 1 month ago

VC applications currently detect black frames by inspecting the payload of each frame. In some cases, the system can provide information that the frames are black due to a camera occlusion. Exposing this via a stat would allow the application to implement the same functionality with lower resource usage.

cc @youennf @jan-ivar

youennf commented 1 month ago

In some cases, the system can provide information that the frames are black due to a camera occlusion.

If some OSes already expose that information, this seems ok to me. The question we could discuss is whether a track should be muted instead in that case.

Another question is how web application would know whether the OS supports it or not. One option would be to have the counter property be undefined for OSes that do not support this. For OSes that expose that information, the counter property would be exposed with 0 as the default value.

eladalon1983 commented 1 month ago

The question we could discuss is whether a track should be muted instead in that case.

I'd say that it's likely helpful to mute in this case (more efficient), *if* we ensure that the app can detect the reason.

guidou commented 1 month ago

If some OSes already expose that information, this seems ok to me. The question we could discuss is whether a track should be muted instead in that case.

Some OSes do for some machines. I think muting would be useful since that would give a signal to the application to stop using the less efficient frame inspection and look at the counter.

Another question is how web application would know whether the OS supports it or not. One option would be to have the counter property be undefined for OSes that do not support this. For OSes that expose that information, the counter property would be exposed with 0 as the default value.

That makes sense to me.

eladalon1983 commented 1 month ago

I think muting would be useful since that would give a signal to the application to stop using the less efficient frame inspection and look at the counter.

Ideally, an app running on a UA that would mute in this case, would not even need to run the less efficient black-frame-detection at all. Maybe we should give the app the ability to detect that.

guidou commented 1 month ago

I think muting would be useful since that would give a signal to the application to stop using the less efficient frame inspection and look at the counter.

Ideally, an app running on a UA that would mute in this case, would not even need to run the less efficient black-frame-detection at all. Maybe we should give the app the ability to detect that.

Correct. @youennf 's idea of not exposing the counter on systems that don't support this would be a way to detect it.

alvestrand commented 1 month ago

Are we sure a browser can always tell if the OS exposes this information or not? Or might it be camera driver specific?

guidou commented 1 month ago

Are we sure a browser can always tell if the OS exposes this information or not? Or might it be camera driver specific?

I think we can expose that the OS can potentially expose this, but I guess it's camera/laptop/driver/somethingelse specific. So this is would not be a full substitute for applications detecting black frames, but a way to do it more efficiently (and perhaps more accurately) in some cases.