Closed wpt-issue-mover closed 6 years ago
In June 2017 discussions in several threads, and also a VC call, discussing Web Bluetooth, Web USB, and WebVR resulted in this summary of specification with matching test APIs. I'll include the summary here:
Related to this thread Specifications with matching test APIs and the earlier thread UserAgent-specific files in Web Platform Tests
This is a summary of a call discussing web platform testing with “Test APIs” including James Graham (Mozilla) and Philip Jägenstedt, Reilly Grant, Vincent Scheib (Google).
Web Bluetooth, WebUSB, and WebVR are actively seeking to develop web platform tests with new requirements compared to existing tests. Complex state must be configured for these features to be tested, e.g. fake bluetooth devices. “Test APIs” are proposed as being paired with specifications.
Concerns raised and discussed include:
A) With new testing patterns we may develop unexpected influence on future implementations. Particularly risky when only a small number of developers are influencing the API. E.g. WebUSB which doesn’t yet have other implementations.
B) WebDriver already operates at a high level, helping avoid test APIs being too implementation specific.
C) Test APIs that don’t work with browser’s standard shipping versions may be a problem. E.g. they won’t run with web-developer focused testing infrastructure, BrowserStack, Sauce, etc. And, they’re harder to run manually.
D) Our primary goal is to have conforming web browser implementations. Considering the goal of “Testing APIs should work for web-app developers” is lower priority.
Recommendations:
Test APIs should clearly indicate their purpose and scope:
They are intended only for WPT. This means test APIs can be modified more freely if a later implementation discovers limitations in the testing API. We do not want additional resistance to API change due to e.g. web app developers using the test API.
Test APIs should be designed carefully to only use concepts from the feature being tested, and not the details of the implementation. E.g. garbage collection, implementation specific feature details, synchronous responses, etc.
Note that some support for automation is currently available and in regular use. /cc @foolip
It's fun to see this old issue and that we've actually resolved half of it now. doNextStepWithUserGesture is now supported. test_driver.bless()
is the shorthand for it, but that's built on test_driver.click()
which you can also use.
We have type:untestable issues for tracking missing capabilities, and there are many. For window.testRunner.setMockScreenOrientation, I think No way to test sensor APIs is the right issue, and @Honry is working on the API in https://github.com/w3c/sensors/pull/369.
For Web Bluetooth, WebUSB, and WebVR, a different form of mocking has emerged using Mojo JS. @Hexcles, is there a tracking page or issue for those capabilities?
Closing this issue now since the basic infrastructure (testdriver.js) is there, but I acknowledge there is much work to be done for specific APIs. Please file new issues for specifics and add the type:untestable label if you have them.
Originally posted as https://github.com/w3c/testharness.js/issues/85 by @scheib on 09 Oct 2014, 22:08 UTC: