Ask for tests for normative changes in CONTRIBUTING.md

[foolip]

See https://github.com/foolip/testing-in-standards/blob/master/policy.md for context. This phrasing matches that used for many other specs' CONTRIBUTING.md files.

[foolip]

Disclaimer: I'm not entirely sure this makes sense or if most changes will just result in a "this is impossible" test issue being filed. But given the state of https://github.com/w3c/web-platform-tests/tree/master/vibration it seemed like maybe it'd work, please discuss :)

[anssiko]

Thanks for the PR. The spec went to Rec with the current tests.

That said, if someone figures out how the test coverage could be reasonably increased, I’d +1 that. Let’s merge this and see if magic happens.

Cc @Honry to get his feedback.

[Honry]

@anssiko, I'm satisfied with current test coverage. :)

[anssiko]

@Honry, thanks for (re-)reviewing the tests.

@foolip, if you have concrete ideas for improvements to the the (inherently) mostly manual tests please let us know.

[foolip]

@anssiko, I see the tests say things like "After hitting the button below, your device must vibrate for a short period of time", which is the reverse problem compared to many input/device specs have.

I wonder what kinds of regression tests exists to make sure that this API doesn't silently stop vibrating because of broken plumbing? If there's nothing like that even without the constraint of being able to share the tests, then maybe this problem is not very tractable.

@JKereliuk FYI, this is a category of manual tests I wasn't really aware of, and perhaps there are more.

[anssiko]

@foolip, to move this discussion off of this PR, is there a generic meta issue somewhere we can subscribe to to get updates on the manual testing best practices for w-p-t?

I guess @Honry who's this group's testing champion is already looped in to these discussion, and will make sure this group's specs follow the w-p-t best practices in general.

[foolip]

@gsnedders, is there anything like that that you know about?

[gsnedders]

@foolip @anssiko No, not really. There's occasional discussion on public-test-infra and there's the type:untestable label on the wpt repo.

[anssiko]

@gsnedders, thanks, then I'm already looped in appropriately.

@foolip, related, @alexshalamov implemented a test runner for manual tests for the purpose of Generic Sensors:

https://intel.github.io/generic-sensor-demos/sensor-tester/build/bundled/ https://github.com/intel/generic-sensor-demos/tree/master/sensor-tester

(Enable chrome://flags/#enable-generic-sensor and chrome://flags/#enable-generic-sensor-extra-classes)

We're thinking whether it'd make sense to try generalize it to run w-p-t tests annotated with <meta name="flags" content="interact">. We could try reuse it for Vibration API.

[foolip]

@anssiko, certainly a generic runner for manual tests would be possible to build, and in fact this is already done for CSS with the old build system and test results reporting. @gsnedders knows the details.

But, I think anything like that will only receive attention as long as the feature is new and interesting, and it can't really scale to regularly run all manual tests that would be interesting to run. That being said, don't let me discourage you from running whatever manual tests are immediately useful to run, manual tests are a little better than no tests after all.

[anssiko]

@foolip, understood. Since the long-term solution that scales is still TBD, we'll try improve the runner for manual tests as a low-hanging fruit, apply it to Vibration API, perhaps Battery Status API after Generic Sensors.