HTML report (--log-html) shows incorrect summary

[galmacky]

Commandline tried: ./wpt run android_webview dom/ --log-html a.html --log-mach - (chrome_android shows a similar result)

a.html (attached) has the following summary: 322 tests ran in 534.9 seconds. 0 passed, 0 skipped, 0 failed, 10 errors. 0 expected failures, 0 unexpected passes, 0 known intermittent results.

This doesn't seem quite right because the number of passed, skipped, failed, and errors don't sum up to the total number of tests ran (322).

I couldn't attach the file, but here's the beginning of the rest of the HTML file:

EXPECTED_OK | /dom/abort/event.any.worker.html | 1.28 |  

EXPECTED_OK | /dom/abort/event.any.html | 0.37 |  

EXPECTED_OK | /dom/collections/HTMLCollection-as-prototype.html | 0.37 |  

EXPECTED_OK | /dom/collections/HTMLCollection-delete.html | 1.18 |  

EXPECTED_OK | /dom/collections/HTMLCollection-empty-name.html | 1.02 |  

EXPECTED_OK | /dom/collections/HTMLCollection-supported-property-indices.html | 0.37 |  

EXPECTED_OK | /dom/collections/HTMLCollection-supported-property-names.html | 1.08 |  

EXPECTED_OK | /dom/collections/domstringmap-supported-property-names.html | 0.87 |  

EXPECTED_OK | /dom/collections/namednodemap-supported-property-names.html | 0.42 |  

EXPECTED_OK | /dom/events/AddEventListenerOptions-once.html | 0.42 |  

EXPECTED_OK | /dom/events/AddEventListenerOptions-passive.html | 0.32 |  

EXPECTED_OK | /dom/events/CustomEvent.html | 0.32 |  

EXPECTED_OK | /dom/events/Event-cancelBubble.html | 0.37 |  

EXPECTED_OK | /dom/events/Event-constants.html | 0.47 |  

EXPECTED_OK | /dom/events/Event-constructors.html | 0.52 |  

EXPECTED_OK | /dom/events/Event-defaultPrevented-after-dispatch.html | 0.37 |  

EXPECTED_OK | /dom/events/Event-defaultPrevented.html | 0.42 |  

EXPECTED_OK | /dom/events/Event-dispatch-bubble-canceled.html | 0.37 |  

EXPECTED_OK | /dom/events/Event-dispatch-bubbles-false.html | 0.37 |  

EXPECTED_OK | /dom/events/Event-dispatch-bubbles-true.html | 0.42 |  

EXPECTED_OK | /dom/events/Event-dispatch-click.html | 0.42 |  

EXPECTED_OK | /dom/events/Event-dispatch-detached-click.html | 0.42 |  

EXPECTED_OK | /dom/events/Event-dispatch-handlers-changed.html | 0.47 |  

EXPECTED_OK | /dom/events/Event-dispatch-listener-order.window.html | 0.87 |  

EXPECTED_OK | /dom/events/Event-dispatch-multiple-cancelBubble.html | 1.02 |  

EXPECTED_OK | /dom/events/Event-dispatch-multiple-stopPropagation.html | 0.37 |  

EXPECTED_OK | /dom/events/Event-dispatch-omitted-capture.html | 0.47 |  

UNEXPECTED_TIMEOUT | /dom/events/Event-dispatch-on-disabled-elements.html | 60.38 |  

EXPECTED_OK | /dom/events/Event-dispatch-order.html | 0.87 |  

EXPECTED_OK | /dom/events/Event-dispatch-other-document.html | 0.42 |  

EXPECTED_OK | /dom/events/Event-dispatch-propagation-stopped.html | 0.37 ...

[galmacky]

@Hexcles

[Hexcles]

This is a general issue related to the HTML log handler (or its interaction with wptrunner). I've tried Firefox & Chrome and the numbers are also wrong in the HTML file.

AFAIK, no one really uses the html report and it must have bit-rotten somewhere...

What do you need it for? The JSON format (--log-wptreport) is much easier to consume.

[galmacky]

Hmm... Ideally speaking, I need the results that can provide me the following: 1) check the overview: I can have an overview at different levels (top level, directory level, test file level) 2) get to the failure stack traces when I want to. 3) access partial results before tests get finished: it takes too long to see any partial results in the form of reporting

log-wptreport seems to provide 1) and 2), but it's a bit inconvenient to use a separate tool to browser JSON file. Also, it seems to provide only two levels of details.

Is there any way to compile everything into HTML page as tests progress, in the format of http://wpt.fyi/results?

On Wed, Aug 14, 2019 at 10:58 AM Robert Ma notifications@github.com wrote:

This is a general issue related to the HTML log handler (or its interaction with wptrunner). I've tried Firefox & Chrome and the numbers are also wrong in the HTML file.

AFAIK, no one really uses the html report and it must have bit-rotten somewhere...

What do you need it for? The JSON format (--log-wptreport) is much easier to consume.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/web-platform-tests/wpt/issues/18408?email_source=notifications&email_token=ABS7TR3M3YOA77BW7HUFRKTQERBVJA5CNFSM4ILOBVEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JTKNQ#issuecomment-521352502, or mute the thread https://github.com/notifications/unsubscribe-auth/ABS7TR3KUAABQOM5PL3WZZTQERBVJANCNFSM4ILOBVEA .

[Hexcles]

Are your needs ranked by importance? I'm afraid we don't have a logger that can satisfy all the needs at the moment.

The default --log-grouped is a human-friendly, streaming log that tells you both the progress and the detail stack trace (2 & 3).

Also, it seems to provide only two levels of details.

Are you saying the JSON does not have stack trace?

[galmacky]

Robert, thanks for the answers.

I might be missing something, but when I try locally --log-grouped seems to produce some errors: $ ./wpt run android_webview dom/ --log-grouped a.txt [0/322] /dom/abort/event.any.worker.html mozlog.structuredlog: Failure calling log handler: Traceback (most recent call last): File "/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/structuredlog.py", line 273, in _handle_log handler(data) File "/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/handlers/base.py", line 74, in call formatted = self.formatter(data) File "/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/handlers/base.py", line 42, in call return self.inner(item) File "/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/reader.py", line 74, in call return handler(data) File "/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/formatters/grouping.py", line 235, in test_end del self.running_tests[data['thread']] KeyError: 'TestRunnerManager-1'

Also, it seems to provide only two levels of details. Are you saying the JSON does not have stack trace?

No, I'm just saying that JSON is just a bunch of nodes data and the top level is Object. Here's an example JSON result:

object {4} run_info {14} product : chrome_android bits : 64 has_sandbox : true linux_distro : Debian verify : false automation : false os_version : rodete version : Debian rodete wasm : false webrender : false debug : false os : linux processor : x86_64 revision : 82629f3a4950e6c2072a3aa993f8d5b52c661c78 time_start : 15599573441272019-06-08T01:29:04.127Z time_end : 15599574519952019-06-08T01:30:51.995Z results [14] 0 {5} 1 {6} 2 {5} 3 {6}

It only has top-level detail and actual test results. It does not seem to produce any subdirectory-level summary, for example.

On Wed, Aug 14, 2019 at 2:31 PM Robert Ma notifications@github.com wrote:

Are your needs ranked by importance? I'm afraid we don't have a logger that can satisfy all the needs at the moment.

The default --log-grouped is a human-friendly, streaming log that tells you both the progress and the detail stack trace (2 & 3).

Also, it seems to provide only two levels of details.

Are you saying the JSON does not have stack trace?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/web-platform-tests/wpt/issues/18408?email_source=notifications&email_token=ABS7TRZ7TYVIL7F4WWYXKADQER2UHA5CNFSM4ILOBVEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KFN2Y#issuecomment-521426667, or mute the thread https://github.com/notifications/unsubscribe-auth/ABS7TRYJ6YG33JL6Y2K66GLQER2UHANCNFSM4ILOBVEA .

[Hexcles]

Hmm, the grouped logger doesn't seem to be able to log to a file. --log-grouped - works fine. @jgraham is this expected?

As for the JSON report, yes the results field of the top-level object is just a list of individual test results without the directory structure. Some post-processing is needed to get the counts you want.