web-platform-tests / rfcs

web-platform-tests RFCs
83 stars 67 forks source link

RFC: Add TestDriver support for the generate_test_report WebDriver extension command. #7

Closed paulmeyer90 closed 5 years ago

paulmeyer90 commented 5 years ago

Summary

I propose to add TestDriver support for the "generate_test_report" WebDriver command, which will allow generic reports to be consistently generated during web platform tests. This command is specced as part of the Reporting API spec here.

Motivation

The real-world causes of reports (like deprecations, interventions, and crashes) can differ between different user agents. In order to test reporting consistently, we need the ability to generate reports universally across the web platform. This command provides that ability.

Risks

I am not aware of any risks.

Details

No additional details.

jgraham commented 5 years ago

This is somewhat off-topic, but I think the way that spec introduces the command isn't what WebDriver intends; it isn't clear what the actual URL for the endpoint is. I don't think you need to define a prefix or anything in this case, just state what the URL is (because this extension isn't a vendor extension).

In any case I think this proposal looks fine, although to be clear that shouldn't be read as any kind of statement on the overall Reporting API spec, which I know nothing about. I assume that were the spec to change, you would keep the tests and testdriver up-to-date with the current spec.

paulmeyer90 commented 5 years ago

Okay, so to clarify, are you suggesting that I just combine the listed Prefix and Name into one URL field?

If not, could you point me to an example of the format you're looking for?

jgraham commented 5 years ago

Okay, so to clarify, are you suggesting that I just combine the listed Prefix and Name into one URL field?

Yes.

paulmeyer90 commented 5 years ago

Reporting spec updated here: https://github.com/w3c/reporting/pull/150

foolip commented 5 years ago

This LGTM, glad you could make it all work together, @paulmeyer90! @LukeZielinski FYI, new automation API incoming.

foolip commented 5 years ago

@paulmeyer90 we've reached "In the case of no substantive disagreement the RFC is considered accepted after 1 week" in the process now. However, https://github.com/web-platform-tests/rfcs/issues/9 suggets doing RFCs in files in the repo instead. Would you be willing to submit a PR with this as a markdown file so we have it for all RFCs? (We just launched this process, tweaking as we go.)

foolip commented 5 years ago

This will be resolved by https://github.com/web-platform-tests/rfcs/pull/13, thanks @paulmeyer90 and @jgraham!