WebKit CI cannot resolve URLs like www.localhost

[youennf]

This issue stems from discussions that started at https://github.com/web-platform-tests/wpt/pull/15191. It is a follow-up to https://github.com/web-platform-tests/wpt/issues/3243.

WebKit is currently not supporting the resolution of URLs like www.localhost or www1.localhost as part of WebKit CI. WebKit is also not requiring WebKit developers to change etc/hosts to do local WPT testing with Safari. common/get-host-info.sub.js allows abstracting this out by using 127.0.0.1 when a cross origin domain is required. This approach works but has some issues:

• It cannot be used everywhere
• It is not as used and as convenient as the WPT templating mechanism, like {{hosts}}[www] or www.{{host}}.

One possibility would be for WebKit people to continue modifying the tests to use get-host-info.sub.js. This has been so far WebKit strategy.

Another possibility might be to update the WPT server templating mechanism so that WebKit could run/update those tests more easily without creating burden on test authors. There are a few possibilities in that area I can think of:

1. Add a configuration option to customize what {{hosts}}[www] would resolve to, or add a new {{cross_origin_host}} keyword with a similar principle. WebKit CI would override the default value.
2. Introduce new template keywords for generating an origin, potentially with an optional port, since tests usually tend to use an origin as the base of an URL resource. https://{{host}}[www]:{{ports[https][0]}} could then be rewritten as {{https_remote_origin}}. With both 1 and 2, test authors could still use whatever they want and WebKit contributors would need to update the tests wherever appropriate.

Thoughts?

[foolip]

I think option 1 is already possible, although it's not easy to find the documentation for how to change the hosts. @jugglinmike has recently done this for web-platform-tests.live, perhaps they can point at the right docs?

On option 2, does that amount to a shorthand for any non-same-origin host? If you don't require /etc/hosts to be changed, what would you replace it with? I'm guessing both 127.0.0.1 and localhost would still be considered same-origin?

[foolip]

@youennf can I assign this to you to resolve one way or another? I came here while triaging the infra label, will mark it as backlog to get it out of my triage queue.

[jugglinmike]

I think option 1 is already possible, although it's not easy to find the documentation for how to change the hosts. @jugglinmike has recently done this for web-platform-tests.live, perhaps they can point at the right docs?

web-platform-tests.live is using wptserve's default configuration for subdomains.

There's a code path which suggests that the names may be overridden, but I haven't been able to find how that is exposed to end users. Even if it is, though, that won't address @youennf's use case. It assumes that the alternate domains are true subdomains. Option 1 is for, e.g. domains[www1] to resolve to another domain entirely (e.g. 127.0.0.1).

It seems like we'd need to make a lot of infrastructure change to support that behavior.

Configuring wptserve like this may invalidate tests. For instance, replacing {{domains[天気の良い日]}} with 127.0.0.1 generally won't be appropriate because those tests most likely concern punycode. That may be acceptable for WebKit's use case, but it may make it difficult to justify adding the functionality to wptserve.

2. Introduce new template keywords for generating an origin, potentially with an optional port, since tests usually tend to use an origin as the base of an URL resource. https://{{host}}[www]:{{ports[https][0]}} could then be rewritten as {{https_remote_origin}}.

We have something like this already: the "sub" feature replaces {{hosts[alt][]}} with the "alternate_host" configuration value.

web-platform-tests.live is configured to use not-web-platform-tests.live as its alternate host. That's why, e.g. http://web-platform-tests.live/cookies/resources/cookie-helper.sub.js includes the following code:

var CROSS_ORIGIN_HOST = "not-web-platform-tests.live";

It seems likely that many tests which use {{domains[www]}} really just need any alternate domain and that they could therefore be re-written to use {{hosts[alt][]}} instead. WebKit devs could configure wptserve to use localhost as the domain and 127.0.0.1 as the alternate domain.

That seems like a more agreeable solution to land here in WPT because it doesn't risk invalidation--those punycode tests will still rely on punycode domains, for instance.

What do you think, @youennf?

[jgraham]

In general I'm wary of unlintable rules that test authors are expected to apply in order to make things work on vendor infrastructure; apart from being difficulty to maintain, it seems like it's passing on costs from the vendor wanting special consideration to all test authors. In this case it seems even worse because there are still a subset of tests that simply won't work for WebKit developers unless they configure things to allow the richer set of hosts.

[jugglinmike]

We would definitely encourage contributors to prefer {{hosts[alt][]}} where possible, but the burden would ultimately fall on WebKit developers. Assuming this solution would even satisfy their use case. Still curious to hear from @youennf about that.

@jgraham do you have any ideas for how we could help out?

[youennf]

I'm guessing both 127.0.0.1 and localhost would still be considered same-origin?

127.0.0.1 and localhost are considered cross-origin.

@youennf can I assign this to you to resolve one way or another? I came here while triaging the infra label, will mark it as backlog to get it out of my triage queue.

Sure, I would really like to come to a conclusion as it causes issues on WebKit side.

It seems likely that many tests which use {{domains[www]}} really just need any alternate domain and that they could therefore be re-written to use {{hosts[alt][]}} instead. WebKit devs could configure wptserve to use localhost as the domain and 127.0.0.1 as the alternate domain.

Agreed.

That seems like a more agreeable solution to land here in WPT because it doesn't risk invalidation--those punycode tests will still rely on punycode domains, for instance.

What do you think, @youennf?

This is very close to option 1. Given that there is already support in WPT infra, I would go with it.

We would definitely encourage contributors to prefer {{hosts[alt][]}} where possible, but the burden would ultimately fall on WebKit developers. Assuming this solution would even satisfy their use case. Still curious to hear from @youennf about that.

I am unsure how you can encourage contributors to prefer {{hosts[alt][]}}. I guess the best encouragement is for existing tests to mostly use {{hosts[alt][]}}. And this task would be on WebKit developers.

Contrary to using get-host-info.sub.js, the changes will be minimal and will create minimal burden on original test authors to re-understand their tests modified by WebKit developers.

[jugglinmike]

Yes, exemplifying the preferred use is one way to encourage it. Another would be to describe it in the project documentation. A third way would be to post to the public-test-infra mailing list.

[youennf]

OK, if everybody agrees, we will start doing this, probably starting with resource-timing and referrer policy tests.

[jgraham]

I don't have a better suggestion because, fundamentally, I don't understand why the need to configure hosts, either in the system or in the test setup for the browser, is an insurmountable problem for WebKit but not for anyone else. Encouraging people to bandaid fixes so that only some tests are broken seems like it's avoiding addressing the fundamental issue here.

[foolip]

if everybody agrees, we will start doing this

@youennf I'm not entirely clear on what the conclusion was here. Is something new in wptserve needed, or will you just replace some pattern in tests with an equivalent and simpler pattern that you can special-case in WebKit?

[youennf]

The plan is to:

• special case {{hosts[alt][]}} to resolve to 127.0.0.1 in WebKit configuration
• replace some use of "{{hosts[][www]}}" or equivalent to {{hosts[alt][]}} This does not require any wptserve change.

I still think that it is a bit tedious to have to write something like https://{{hosts[alt][]}}:{{ports[https][0]}}. But this could be left to another day.

[foolip]

@youennf if "{{hosts[][www]}}" doesn't work currently, could you map that to 127.0.0.1 instead? It couldn't work less than it currently does, right?

Switching from "{{hosts[][www]}}" to "{{hosts[alt][]}}" would switch from a same-site (for cookies) to a non-same-site host. For many tests it won't matter, but if you're going to map something 127.0.0.1, isn't the behavior of "{{hosts[][www]}}" closer to the actual behavior you'll get? (Cookies will be shared.)

[youennf]

Currently, WebKit has no way to run tests relying on hierarchical URLS, which is useful for some tests like cookies. These tests should remain as is. As pointed out by @jgraham, I hope we can work on a solution on WebKit side to run those tests.

In the meantime, WebKit can benefit of running those other tests that do not care about differences between hosts[alt][] or hosts[][www] as long as it is cross origin. These are the tests we would change and I do not see any big downside in doing that.

I actually see a benefit in that effort. Currently, it is not clear whether a test is using a particular port or a particular prefix for specific port/prefix reasons or just for being cross origin. If we can replace all these give-me-a-cross-origin-url cases with one http/https way, the intent of tests using specific port/prefix might be easier to understand.

[foolip]

@youennf did you end up sending any PRs to rewrite some tests?

[annevk]

Thanks @domfarolino for pointing me here.

This effort is a little concerning to me as there are many Fetch tests that do care about the distinction between cross-origin and cross-site and I'm worried that this will lead to accidental regressions. And it also doesn't really help as WebKit will still need to create some infrastructure to test cross-origin (but same-site).

[domfarolino]

To answer @foolip, yes it seems that at least https://github.com/web-platform-tests/wpt/pull/16568 and I think some others have been uploaded, however I think it is safe to block them on resolution in this thread, as the effort seems controversial.