search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.98k stars 3.09k forks source link

Issues with signal detection in RTCPeerConnection-helper.js #23340

Open foolip opened 4 years ago

foolip commented 4 years ago

This issue is to track two things from https://github.com/web-platform-tests/wpt/pull/22779.

  1. Using OffscreenCanvas works, but it's not enabled in Firefox or Safari yet, so the tests will fail because of that. It's probably possible to use a canvas element instead, with a TODO to replace it later.
  2. The signal detection allows only for an error of 1, but due to quantization error in video codecs this could probably be larger. The margin of error needs to be large enough, but not so large that anything will pass.

Alternative idea for signal: send a specific color like 00FF00 green, and detect a signal if it's "almost" green. This should allow for a bigger margin without risking false detection, as long as there's nothing almost green elsewhere in the frame.

See https://github.com/web-platform-tests/wpt/pull/22779#discussion_r416513876 for original discussion.

alvestrand commented 4 years ago

On signal: humans are usually seen to be better able to percieve luma (brightness) than color - so codecs tend to optimize for luma fidelity. It's been suggested that instead of taking the average across the R/G/B values, we should use the ITU luma formula:

  // Use Luma as in Rec. 709: Y′709 = 0.21R + 0.72G + 0.07B;
  luma += 0.21 * data[i] + 0.72 * data[i + 1] + 0.07 * data[i + 2];

(code from https://github.com/webrtc/testrtc/blob/master/src/js/videoframechecker.js) This way, shakiness in the blue would have almost no effect on the output signal.

But the easy thing to do to allay the concern is to write a test for the framework that scans through a range of "signal" values for a given and sees if it can reliably detect the changes; if there are ranges of values where detection doesn't work, we should prevent them from being used.