[eme] Differentiate between unsupported features and successful test execution

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others

https://web-platform-tests.org/

Other

4.89k stars 3.06k forks source link

[eme] Differentiate between unsupported features and successful test execution #3765

Open ddorwin opened 7 years ago

ddorwin commented 7 years ago

In some of the tests, especially negative tests, not supporting a feature may currently be considered a pass whereas that does not actually demonstrate correct implementation of the feature under test.

This is definitely true of the *-generate-request-disallowed-input.html tests and may be true for others. This contributed to the hiding of test bug #3763 where none of those tests were being run as intended because "not supported" was considered a success.

ddorwin commented 7 years ago

Perhaps a result of NOTRUN would be appropriate for such cases.

There are currently problems with this results, though, as in #3735:

NOTRUN can only be returned if test.done() is called without any assertions being tested.
- Not asserting would be difficult to maintain and could prevent some hardening mechanisms (#3764).
NOTRUN appears as a timeout in the test report.

mwatson2 commented 7 years ago

In #3763 I was not able to get NOTRUN to be returned for individual tests on the page. I think this leaves us with the option of simply not running the tests for unsupported parameter combinations, so no results for those test names will appear in the final test results.

louaybassbouss commented 7 years ago

I think it is important to ensure that all tests are displayed in the report. Example: let say you have 100 tests and Browser A runs 80 tests and Browser B runs 70 (subset of the 80) in the test report 20 tests will not be displayed.

louaybassbouss commented 7 years ago

@ddorwin NOTRUN background color in the same as for TIMEOUT (orange) in the test report but the content is still different.

mwatson2 commented 7 years ago

I'm not sure we have a good way to do this in the current test framework. I suggest for the moment that such tests simply return TIMEOUT (as for various other failures).