How to handle mock or automated behavior?

[wpt-issue-mover]

Originally posted as https://github.com/w3c/testharness.js/issues/85 by @scheib on 09 Oct 2014, 22:08 UTC:

Some tests require the platform or user to be mock-ed or automated for tests to run on automated infrastructure. For example, Chromium LayoutTests have access to TestRunner which exposes methods such as window.testRunner.setMockScreenOrientation.

However, without care these tests will not function when used in other browsers, or with manual testing. One example are the Chromium Pointer Lock tests using doNextStepWithUserGesture which allow automation to mock a user-gesture (e.g. mouse click/key press) but also provide a way for manual testers to progress tests.

Has design been discussed to extend testharness with a mock concept that browsers can populate and which when possible falls through to manual instructions?

[scheib]

In June 2017 discussions in several threads, and also a VC call, discussing Web Bluetooth, Web USB, and WebVR resulted in this summary of specification with matching test APIs. I'll include the summary here:

Related to this thread Specifications with matching test APIs and the earlier thread UserAgent-specific files in Web Platform Tests

This is a summary of a call discussing web platform testing with “Test APIs” including James Graham (Mozilla) and Philip Jägenstedt, Reilly Grant, Vincent Scheib (Google).

Web Bluetooth, WebUSB, and WebVR are actively seeking to develop web platform tests with new requirements compared to existing tests. Complex state must be configured for these features to be tested, e.g. fake bluetooth devices. “Test APIs” are proposed as being paired with specifications.

Concerns raised and discussed include:

A) With new testing patterns we may develop unexpected influence on future implementations. Particularly risky when only a small number of developers are influencing the API. E.g. WebUSB which doesn’t yet have other implementations.

B) WebDriver already operates at a high level, helping avoid test APIs being too implementation specific.

C) Test APIs that don’t work with browser’s standard shipping versions may be a problem. E.g. they won’t run with web-developer focused testing infrastructure, BrowserStack, Sauce, etc. And, they’re harder to run manually.

D) Our primary goal is to have conforming web browser implementations. Considering the goal of “Testing APIs should work for web-app developers” is lower priority.

Recommendations:

Test APIs should clearly indicate their purpose and scope:

• They are intended only for WPT. This means test APIs can be modified more freely if a later implementation discovers limitations in the testing API. We do not want additional resistance to API change due to e.g. web app developers using the test API.

• Test APIs should be designed carefully to only use concepts from the feature being tested, and not the details of the implementation. E.g. garbage collection, implementation specific feature details, synchronous responses, etc.

[RByers]

Note that some support for automation is currently available and in regular use. /cc @foolip

[foolip]

It's fun to see this old issue and that we've actually resolved half of it now. doNextStepWithUserGesture is now supported. test_driver.bless() is the shorthand for it, but that's built on test_driver.click() which you can also use.

We have type:untestable issues for tracking missing capabilities, and there are many. For window.testRunner.setMockScreenOrientation, I think No way to test sensor APIs is the right issue, and @Honry is working on the API in https://github.com/w3c/sensors/pull/369.

For Web Bluetooth, WebUSB, and WebVR, a different form of mocking has emerged using Mojo JS. @Hexcles, is there a tracking page or issue for those capabilities?

Closing this issue now since the basic infrastructure (testdriver.js) is there, but I acknowledge there is much work to be done for specific APIs. Please file new issues for specifics and add the type:untestable label if you have them.