web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
5k stars 3.1k forks source link

Substitution in headers and body doesn't retain context. #20933

Open mikewest opened 4 years ago

mikewest commented 4 years ago

Because cookie tests that work on hard-coded names stomp all over each other if run in parallel, it would be nice to somehow generate something unique for a given test.

One way of attacking that problem is to run some JavaScript that sets a randomly-named cookie, or asks the server to do the same.

It would be nice if we could use the existing .headers mechanism to set headers directly when loading a test, rather than hopping back to the server via fetch(). I poked at this a bit, but it doesn't seem to work. I'd hoped that something like the following would work:

amazing-test.sub.html.sub.headers

Set-Cookie: name{{$name:uuid()}}=value; ...
amazing-test.sub.html

...
do_some_amazing_testing_on_cookie_named("name{{$name}}");
...

Unfortunately that dies with a templating error (the binding from the headers file isn't persisted into the test's body, and vice versa. Is that something that we could change?

Alternatively, is there a clever way to generate a random-but-unique identifier for the resource that's currently loading that could be available in both places?

foolip commented 4 years ago

In https://web-platform-tests.org/writing-tests/server-features.html#tests-requiring-special-headers there's no hint that this is supported, however here it looks like it should work: https://github.com/web-platform-tests/wpt/blob/d858c038e3056c20d791089484e6b61a2fbc939d/tools/wptserve/wptserve/handlers.py#L174-L180

Looks like @jgraham wrote that code.

foolip commented 4 years ago

@mikewest maybe you already got that far, and all that's missing is some syntax to generate a random value?

mikewest commented 4 years ago

maybe you already got that far, and all that's missing is some syntax to generate a random value?

I did get that far! Both .sub.html and .sub.html.sub.headers work just fine with regard to causing substitution, and {{uuid()}} creates a random value. The issue is that {{uuid()}} creates a different random value on each execution. I hoped that the {{$name:uuid()}} syntax in one file would create a binding between the specific random value it created in the header file, and {{$name}} in the test file. That doesn't work, as {{$name}} isn't a bound variable when substitution is performed on the second file.