Add way to analyze reftests from logging output

[jgraham]

At the moment to analyze reftest failures one either needs to run with --log-tbpl or --log-raw and paste the result into the relevant reftest analyzer from mozilla-central. But that is highly non-obvious in two ways; it's both unclear that you need to switch log formats and that there is a tool for analyzing the results.

There are several possible solutions here. One might be to encode all the relevant logging in a (very long) url and link to the reftest analyzer in the log output. So we might have

INFO: To analyze this failure visit https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer-structured.xhtml?logdata=<long string>

An alternative would be to land the analyzer directly in wpt, although I think there is a licensing impediment to that.

[jgraham]

cc @alijuma

[alijuma]

cc @boazsender

[gsnedders]

@jgraham Can we at least document how to do this today somewhere?

[gsnedders]

(Like, I've just spent about ten minutes trying to figure this out. Can you just write down how to do ti today somewhere?)

[gsnedders]

Loading it from https://hg.mozilla.org/ no longer works, as CSP prohibits script execution there.

Googling for "Reftest analyzer" doesn't find anything useful, so I guess the only way to use the analyzer now is "checkout mozilla-central"? Surely something must exist somewhere for Treeherder at least?

[gsnedders]

OK, so the status is the structured reftest logger doesn't work due to CSP prohibiting script execution on hg.mo. The unstructured reftest logger doesn't work because it can't parse the current mozlog output.

[gsnedders]

@jgraham if you want to fix the unstructured logger case, here's a test case:

    { "name": "wpt reftests (Linux local)",
      "log": `SUITE-START | Running 1 tests
TEST-START | /css/css-backgrounds/border-image-6.html
TEST-UNEXPECTED-FAIL | /css/css-backgrounds/border-image-6.html | Testing http://web-platform.test:8000/css/css-backgrounds/border-image-6.html == http://web-platform.test:8000/css/css-backgrounds/border-image-6-ref.html
REFTEST   IMAGE 1 (TEST): data:image/png;base64,
REFTEST   IMAGE 2 (REFERENCE): data:image/png;base64,
TEST-INFO took 425ms
SUITE-END | took 2s`,
      "expected": { "pass": 0, "unexpected": 1, "random": 0, "skip": 0 },
    },

this fails because we don't have REFTEST or INFO or ERROR in the line.

[gsnedders]

https://bugzilla.mozilla.org/show_bug.cgi?id=1530002 and https://bugzilla.mozilla.org/show_bug.cgi?id=1530008 filed for the two issues, FWIW.

[fred-wang]

cc @rwlbuis @bkardell

[jgraham]

Now that we have reftest analyzer functionality in wpt.fyi there might be some way to provide a url to invoke that directly (could involve supporting having the wpt.fyi url contain the image directly, if that's not already possible). @Hexcles

[foolip]

What are the options here? We now have a --log-wptscreenshot argument documented in https://web-platform-tests.org/running-tests/from-local-system.html#output-formats, so some tooling could be built around that.

IMHO, the best user experience would be to save reftest screenshots by default, and be able to use something like the reftest analyzer locally. It shouldn't require special log arguments and stuff that only wpt infra folks would know.