Define a WebDriver extension for Signed Exchange testing

[irori]

Both Signed Exchange and CBOR cert chain have short lifetime (max 7 days). This makes it difficult to use pre-generated signed exchange / cert files in end-to-end testing. To workaround this, Chromium has an internal testing API (available only in test runner) that overrides signature verification time.

In order to test Signed Exchange Loading in web-platform-tests, we need an interoperable way to override signature verification time. The preferred way is defining a WebDriver extension in the respective spec (in this case, Signed Exchange) so that the same surface API is used across UAs.

See this doc for more details and discussions.

[RByers]

@kereliuk, can you point to an example spec where this has been done now? This seems like it should be pretty straight forward to implement, and I'm a little concerned that the longer we continue with chromium-only testing, the more debt we might accidentally accrue.

FWIW it looks like the existing chromium tests are here: third_party/WebKit/LayoutTests/http/tests/loading/htxg third_party/WebKit/LayoutTests/http/tests/devtools/htxg/ content/test/data/htxg/

And there's already a devtools protocol API for mocking out the verification time: setSignedExchangeVerificationTime

Anything I missed @irori?

[kereliuk]

@kereliuk, can you point to an example spec where this has been done now?

Here is an example in Reporting

@irori Let me know if you have any questions with defining the specification or the implementation steps that follow. I'm working on getting some end to end documentation/tutorials on the entire process.

[irori]

I initially thought we can override the verification time, but it seems not an option because OS/underlying library doesn't allow this.

Pasting @sleevi's comment (from an internal chat) about Signed Exchange testing (markup mine):

Here's things we can and can't do:

1. We can modify the HXTG interaction to support treating cert errors as interstitials, so that DevTools can bypass the interstitial (as it does today) - but that's going to be really weird with the whole flow and expected behaviours.

2. We can modify HXTG's cert verifier to lie (similar to --ignore-cert-errors-spki-list), but then like ICESL, we'd need to make sure it doesn't cache (which is why that flag requires --user-data-dir), but that means you can only DevTools with a custom profile (and ICESL already 'just works' for the cert part today)

3. We can generate an on-the-fly CA and on-the-fly responses. It needs to be an on-the-fly CA, not shared among WPT, because our TLS policies are such that we block vendors who generate shared-root-CA keys (meaning we'd blacklist the WPT CA key if it's generating a shared-key). That's the path WebPageReplay has gone, AIUI.

What we can't do:

• We can't verify a certificate and response "in the past" while still actually validating it. No one's APIs support that use case. PKI assumes time is linear and monotonic, and so to do the APIs

From Chromium's point of view, 2. seems to be the easiest. Our test runner already uses --ignore-cert-errors-spki-list for WPT tests, so we just have to let SXG signature verifier lie (about signature expiration) too. Question is if other UAs have similar switches, and if we can/should make this switchable via a WebDriver extension. As Ryan noted, we have to make sure it doesn't cache, etc.

Option 1 (let DevTools bypass interstitial SXG errors) is another way to bypass cert verification in tests. This might be more suitable for an underlying implementation of a WebDriver extension (I'm not completely sure). It's still question if other browsers are willing to implement the same WebDriver API.

Option 3 actually exercises cert verification in tests. In addition to WPT infra work, this needs some Chromium-side testing infra change too (not to just ignore cert errors).

Strawman proposal

• Add .tentative tests using the option 2. It wouldn't work unless UA is configured to ignore certificate errors, and wouldn't test the certificate verification behavior. But it would be still useful to test the "loading" aspect of SXGs. Also, we can add manual tests with instructions to generate SXGs using a test CA.

• In the long term, explore an on-the-fly CA + SXG generation approach. Probably we can implement it in Go and submit binaries to the WPT repository.

[irori]

FYI, I've started adding tentative tests in wpt/signed-exchange. (Tracking bug)

[irori]

We've implemented the option 2 of https://github.com/WICG/webpackage/issues/188#issuecomment-412729051. Probably we won't need a WebDriver extension. Closing.