search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.82k stars 3.01k forks source link

Test statistics are not cleared after suite_end #37415

Open WeizhongX opened 1 year ago

WeizhongX commented 1 year ago

Run "wpt run --repeat=2 chrome badging ...", we get output as below. The total test count, and run expected count are accumulated for the second run. This would also be true for other counts if they present. This is confusing. Can we clean up test statistics once suite_end is called?

Running 4 tests in web-platform-tests

Ran 4 tests finished in 35.9 seconds. • 4 ran as expected. 0 tests skipped.

Running 4 tests in web-platform-tests

Ran 8 tests finished in 9.4 seconds. • 8 ran as expected. 0 tests skipped.

WeizhongX commented 1 year ago

@jgraham @foolip @jonathan-j-lee

jgraham commented 1 year ago

I think having some part of the output show the actual total number of tests that have been run makes sense, but certianly the "8 tests in 9.4 seconds" part is wrong.

WeizhongX commented 1 year ago

Thanks for the comment. I agree having some way for all the tests run could be useful. The thing is we do not have a concept for such runs. The above log is printed when we call "logger.suite_end", so supposedly it should only print out statistics from last "logger.suite_start".

Adding a new concept to capture all the statistics in a single launch of wptrunner could be a big change. Can we do something in the middle? For example, "logger.suite_start" accepts a name parameter, can we only accumulate the statistics when the name matches?

When there are unexpected results, the output could get even messier. All the unexpected results will be printed out multiple times in subsequent repeats or retries, as below:

Ran 3087 tests finished in 12.4 seconds. • 3075 ran as expected. 120 tests skipped. • 1 known intermittent results. • 2 tests had errors unexpectedly • 3 tests passed unexpectedly • 3 tests unexpectedly okay • 7 tests had unexpected subtest results

Tests with unexpected results: Unexpected subtest result in test-a.html

Ran 3152 tests finished in 12.4 seconds. • 3137 ran as expected. 180 tests skipped. • 1 known intermittent results. • 3 tests had errors unexpectedly • 3 tests passed unexpectedly • 3 tests unexpectedly okay • 9 tests had unexpected subtest results

Tests with unexpected results: Unexpected subtest result in test-a.html

Unexpected subtest result in test-a.html

WeizhongX commented 1 year ago

@jgraham, pls advise what is the best way to improve this.

If necessary, we can work on the code change. (I understand we need make code change to Firefox side repo, then release the library?)

jonathan-j-lee commented 1 year ago

I added this workaround in Chromium for now.