Conformance criteria incompatible with standards-track

[jan-ivar]

§ 2. Conformance says: "This specification does not define what objects a conforming implementation should generate. Specifications that refer to this specification have the need to [SIC] specify conformance."

Typo aside, this seems incompatible with our charter, which stipulates that, to exit CR "each normative specification is expected to have two independent implementations of every feature defined in the specification".

Since this spec is in CR, I think this should be updated to concentrate on this spec's conformance criteria, as stipulated by our charter.

[vr000m]

Related to #595.

[alvestrand]

Two different issues here:

• What specs that refer to this have to do about defining their own conformance
• What this spec should do about showing implementation.

If a spec says that stat 1 is mandatory, browser A implements stats 1 and 2, browser B implement stats 1 and 3, and browser C implements 2 and 3, all stats have been implemented twice (this spec is OK), but only two browsers are conformant to that other spec.

[henbos]

I would like webrtc-stats to contain all metrics that have at least one browser implementation. I think it makes perfect sense that we added "at risk" notes after some of the recently added audio metrics became controversial. Hopefully such notes will be rare to see in this document.

Right now there are a small number of metrics in webrtc-provisional-stats that have been implemented in Chromium like contentType based on the video-content-type header extension that I would rather move to webrtc-stats with an "at risk" note.

As for most other metrics (except for a couple of other examples), the https://w3c.github.io/webrtc-provisional-stats/ doc is currently a graveyard of never-implemented and no-interest-to-ever-implement metrics, so I'd rather see us sunset that document in favor of one stats spec to rule them all.

[henbos]

Takeaways from WG editors meeting IIUC: "mandatory to implement" simply refers to the metrics needed in order to be considered conformant to the webrtc-pc spec, a.k.a. "WebRTC 1.0". But the goal of getStats() should of course also be conformance, so in order to be conformant to all of webrtc-stats the user agent should implement all of its metrics. If they're useful, they're useful, and we're no longer shipping any goog-metrics.

So the action item here IMO should be to update the conformance section of webrtc-stats with a PR that removes any language that makes it sound like any metrics are optional. The sentence that says that the implementation "may support generating other stats" too seems like a nonsense statement if our goal is conformance. Which it should be.

[henbos]

Speaking of conformance, does Firefox (@jan-ivar) and Safari (@youennf) have plans to make supported-stats.https.html a greener place? (Also why does Edge not run these tests on the wpt.fyi page? @aboba )

[fippo]

Presumably the Edge bots don't have a camera and WPT continues to insist on not using the good old --use-fake-device-for-media-stream flag?

[dontcallmedom]

re WPT & edge, I had patched it to make use of the flag some time ago: https://github.com/web-platform-tests/wpt/pull/31938/files - but it doesn't look like it sufficed