Integrate manual tests for URL display?

[estark37]

Chromium maintains a tool called Trickuri (pronounced "trickery") for testing how web browsers display URLs (https://github.com/chromium/trickuri) and I'm wondering if there would be interest in having this tool be part of web-platform-tests.

Trickuri is a proxy that serves various unusual URLs (e.g., URLs with super long domains, domains with RTL labels, etc.). The intended usage of Truckuri is to load each unusual URL in a web browser and check how it is displayed in the address bar, and then, on each unusual URL, load a test case for each web platform feature that triggers a browser UI that displays a URL. For example, there's a test case that displays an alert() dialog (to see how the page origin is displayed in the alert), a test case that displays a Credential Manager dialog, etc. The Trickuri user is intended to walk through these test cases and check that each of these web platform features handles each unusual URL properly. Common bugs include eliding URLs improperly (displaying https://victim.com.evil.com as https://victim.com...), not decoding IDNs properly, displaying URLs in a font that lends itself to spoofing (examp1e.com looks very similar to example.com), etc. Unfortunately, the testing is fully manual as we can't really automatically verify that URLs are displayed correctly, but we still find it useful for systematically reviewing how various browser UIs handle unusual URLs.

Would there be interest in having Trickuri be part of web-platform-tests? I'm thinking that this could be beneficial in two ways: 1.) It would be beneficial for browsers to collaborate on the set of unusual URLs that their UIs should be tested against (I'm sure our current set of unusual URLs is not comprehensive), 2.) It would help ensure that new web platform features which display URLs in browser UI get thoroughly tested for common URL display bugs. We could try to make sure that every relevant web platform feature has a test case in Trickuri, but it would be far preferable to integrate into the spec development process as part of web-platform-tests.

[foolip]

@estark37 thanks for filing this issue, interesting stuff!

A directory of manual tests in web-platform-tests is definitely possible. In your description, the only thing that stands out as possibly problematic is the domains themselves, "super long domains, domains with RTL labels, etc." It looks like Trickuri depends on proxy configuration, I'm guessing the proxy also resolves DNS and that's why /etc/hosts need not be modified?

The setup in WPT is a custom web server with its own certificate. The certificate needs to be trusted and we require /etc/hosts to be modified. Adding domains is technically pretty straightforward, but very weird domains might make it look scary/suspect to new contributors. Can you list the domains required? Would it be acceptable that they all end in web-platform.test, or are many unique domains across TLDs required?

cc @annevk who maintains https://url.spec.whatwg.org/ and might have thoughts.

[annevk]

It sounds terrific to have (shared) tests for this. As noted, I think the main problem here will be the infrastructure.

We have some other tests that cannot be written due to lack of flexibility in domain names (e.g., #13465), but it's also true that the required setup for those kind of tests could raise the bar for the remainder of the tests too much.

Perhaps a temporary measure would be a standalone repository. We'd lose a lot of the benefits, but at least there's a shared space to collaborate while there's no suitable infrastructure in WPT.

[estark37]

Thanks for the replies @annevk and @foolip! And doh, I completely lost track of this issue and forgot to follow up. :(

I agree that the current wpt server setup is not a good fit for this. We test all sorts of different domains and I anticipate that new ones will be added fairly regularly, so modifying /etc/hosts doesn't seem viable.

I was imagining that the URL display tests would be sort of a separate workflow, so that the setup doesn't affect the other tests. For example, I was thinking that you would run wpt like normal to run all the automated tests; separately, you could run ./wpt run_url_proxy product (or something) to run product configured with wpt as a proxy server and it steps you through the manual tests. Even though it would be sort of separate, I see it as more likely that the URL display tests would stay up-to-date if they live with wpt, because a spec author would only have to think about adding tests to one repository rather than two. (Perhaps it would also be possible to use the proxy mode to write some of the tests that can't currently exist that @annevk pointed to?)

But, if it seems too weird to have a separate mode for running URL display tests, I'd be fine to keep trickuri in its current home and we will certainly try to keep it up-to-date with wpt.

[annevk]

Thanks, that makes sense to me.

@jgraham @gsnedders your input is probably needed at this point.

[foolip]

A variant of ./wpt run that requires more setup and lets the user run manual tests and say whether they passed sounds like a good idea to me. Right now I think http://w3c-test.org/tools/runner/index.html is the only way to run manual tests.

[foolip]

Ping @gsnedders on the "./wpt run_url_proxy product (or something)" idea.