Should Edge Canary have the 'experimental' label?

[stephenmcgruer]

Forked from https://github.com/web-platform-tests/wpt.fyi/issues/1539. Copying some comments over:

https://github.com/web-platform-tests/wpt.fyi/issues/1539#issuecomment-552049114 "Do we have an issue for the fact that Edge Canary doesn't seem to be tagged with experimental? (Or is there a valid reason it shouldn't be)."

https://github.com/web-platform-tests/wpt.fyi/issues/1539#issuecomment-552073962 "There’s no issue for that AFAIK, and I don’t recall how it came about. @mustjab do you know why it’s Edge Dev and not Canary that shows on wpt.fyi by default?"

https://github.com/web-platform-tests/wpt.fyi/issues/1539#issuecomment-552133209 "The code reason, incidentally, would seem to be that results-processor/wptreport.py has a CHANNEL_TO_LABEL map that doesn't include 'canary' (it maps ['release', 'stable'] --> 'stable', ['beta'] --> 'beta', and ['dev', 'experimental', 'nightly', 'preview'] --> 'experimental'), and that the report from azure doesn't include 'experimental' either."

Assigning to mustjab to answer the query about whether Edge Canary should have the experimental label.

[mustjab]

Yes, don't see any reason why not. Let me know if I need to make any change on our end to set it correctly.

[foolip]

Sounds like we should do it, but also at the same time stop applying the label to Edge Dev. https://github.com/web-platform-tests/wpt.fyi/issues/1575 probably needs to be fixed first though so that one can find older Dev and Canary runs without knowing when the experimental label flipped.

https://github.com/web-platform-tests/wpt.fyi/issues/1519 is also tangentially related.

[stephenmcgruer]

I agree; marking this blocked on https://github.com/web-platform-tests/wpt.fyi/issues/1575

[Hexcles]

Sorry for being late to this series of label-related issues. IIRC, we decided to have Edge Dev on the home page by default because we do not have Chrome Canary on Linux and we want a more aligned comparison.

The code for assigning the experimental label in the processor is technically correct because, incidentally, Edge Canary does not have experimental web platform features enabled. We should fix that and start labelling new Edge Canary runs as experimental, but we probably shouldn't go back and "fix" the old runs -- that'd skew our definition of "experimental".

[foolip]

We're still running both Edge Dev and Canary every 3 hours, but this is pretty wasteful since we don't really use the Canary results. If we label Canary as experimental then we can probably just stop running Edge Dev.

@KyleJu WDYT?

[mwjacksonmsft]

I realize this is an old issue - but I'm happy to take a look at fixing it. With this change https://github.com/web-platform-tests/wpt/pull/42008, we do enable experimental web platform features (amongst other things).

The Edge tests are dependent upon MojoJS, which are only produced from Linux builds, and on Linux we only support dev, beta, and stable channels. If we were to run canary builds, we'd end up with mismatched MojoJS causing test failures. Additionally, I think we probably want to stick with dev channel, to help reduce noise as we ingest changes from Chromium. I think running canary channel today, would make it hard for us to reason about the different failures.

I'd propose that we leave Edge Dev labelled experimental, and Edge Canary as nightly as it is today.

If any of the above changes, then we could revisit. WDYT?