search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.93k stars 3.07k forks source link

Add automation for updating webgpu testsuite #24699

Open jdm opened 4 years ago

jdm commented 4 years ago

https://github.com/web-platform-tests/wpt/tree/master/webgpu#webgpu-conformance-test-suite mentions that the tests are generated from another repository, but doesn't explain how to do that. Now that other browsers are implementing WebGPU (independently, Firefox and Servo in particular), it would be preferable if we didn't have to wait for someone on the Chromium team to perform an update so we could get access to changes like https://github.com/gpuweb/cts/commit/4cd0577a7e97929c357f4446831d514365135ef9.

jdm commented 4 years ago

@austinEng You seem to have performed previous WebGPU CTS rolls in the past. Is it feasible to adapt https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/web_tests/webgpu/roll_webgpu_cts.sh;l=1?q=oll_webgpu_cts.sh%20&sq=&ss=chromium&originalUrl=https:%2F%2Fcs.chromium.org%2F as a more general automation for making a PR to web-platform-tests/wpt, independent of any particular browser's import/export mechanism?

jdm commented 4 years ago

cc @kunalmohan

austinEng commented 4 years ago

I think it would definitely be possible. I think the only Chromium specific bits are the source and destination directories. fyi: @kainino0x

kainino0x commented 4 years ago

see also: #23693

Sorry that I haven't spent any time on this issue. The reason I haven't tried to tackle this is I still want to dig into whether we can move the source code into WPT. But we should continue punting on that and try to find a better solution here.

There's very little to the Chromium roll script - the meat of the generation is inside the gpuweb/cts project, and the only important remaining bit is the rsync invocation. We can probably fairly easily have a (Node) script in gpuweb/cts that can roll itself into a checkout of wpt given just the path to wpt.

The other issue is that these tests are extremely unstable. They change in such a way that browsers stop passing them on a daily basis, due to changes in the upstream spec. I don't know how to prevent this from being a burden on the folks who maintain auto-imports into browsers. But obviously we need to do this somehow - we've avoided this so far by (afaik) chromium being the only browser that tries to care about these tests in their synced version of wpt. And now, we don't even run them - we have another copy of a different branch of gpuweb/cts that we actually run.

I think we're getting quite close (~weeks) to a point where that branch (glsl-dependent) will go away, but we probably don't need to worry about it here since Chromium can manage it separate copy without worrying wpt.

kainino0x commented 4 years ago

Do firefox and servo have "internal" wpt tests like Chromium - tests which run using the wpt harness but are not exported to upstream wpt? If so, they could maintain copies of whatever version they want and completely avoid autoimporting headaches. Maybe we could even delete the copy from wpt until the API gets more stable?

jdm commented 4 years ago

Servo certainly does. It's not entirely clear to me why that's preferable compared to a single shared source of truth, though.

kainino0x commented 4 years ago

Because the WebGPU CTS tests are constantly having breaking changes, and we can't expect all implementations to stay exactly in lock step with one another (not to mention the test developers). It would make a lot more sense for each browser to control its exact version dependency on the tests.

kainino0x commented 4 years ago

@jdm I was about to come back here and ask if you had any feedback on that approach - looks like you've already started working on it! Do you anticipate this approach will work well for you?

cc @kvark @jdashg @litherum @grorg what do you think of the suggestion above: have each browser vendor its copy of the CTS build, so that we can work toward conformance unhindered by trying to synchronize tests through this repo? And possibly deleting the copy inside this repo until the API is close to stable?

jdm commented 4 years ago

I do not anticipate any issues vendoring the testsuite in Servo's repository.

kvark commented 4 years ago

We are going to be vendoring the testsuite separately in Gecko as well. There doesn't seem to be any other way around this.

kainino0x commented 4 years ago

I agree, and thanks for the info. If there are no objections, I'm intending to delete WPT's copy, for now, during our next update in Chromium. Should avoid confusion in the various browsers between the wpt version and the vendored version.