Add RFC for `crash` test type

[jgraham]

Add a test type that just ensures that a page can be loaded without error.

[jgraham]

@emilio I think you were interested in this. If you have feedback it would be great, particularly if you have evidence that sharing load-type tests (e.g. gecko crashtests) between engines is effective at finding issues.

[jugglinmike]

Crashes and other issues caught by these tests may not be sufficiently common between browsers to make sharing these tests a good use of resources.

This was my concern while reading the proposal, so I'm glad you included it! Personally, I'm pessimistic that tests like these do provide much value between implementations. I'm no implementer, though, so I'm very curious about what @emilio has to say on the subject.

[emilio]

I have actually some data on this, need to write it down, but this are the results thus far (I haven't filed all bugs yet) from running Gecko's crashtest suite in WebKit and Chrome, and Chrome's crashtests in Gecko:

• https://bugs.chromium.org/p/chromium/issues/detail?id=1009830
• https://bugs.chromium.org/p/chromium/issues/detail?id=1009827
• https://bugs.chromium.org/p/chromium/issues/detail?id=1009825
• https://bugs.chromium.org/p/chromium/issues/detail?id=1009821
• https://bugs.chromium.org/p/chromium/issues/detail?id=1009817
• https://bugs.chromium.org/p/chromium/issues/detail?id=1009806
• https://bugzilla.mozilla.org/show_bug.cgi?id=1585226
• https://bugzilla.mozilla.org/show_bug.cgi?id=1585303
• https://bugs.webkit.org/show_bug.cgi?id=202803
• https://bugs.webkit.org/show_bug.cgi?id=202804
• https://bugs.webkit.org/show_bug.cgi?id=202805
• https://bugs.webkit.org/show_bug.cgi?id=202807
• https://bugs.webkit.org/show_bug.cgi?id=202809
• https://bugs.webkit.org/show_bug.cgi?id=202811
• https://bugs.webkit.org/show_bug.cgi?id=202812

And as I said I'm still not done running tests and filing bugs. Pending stuff:

• Running Chromium's crashtests on WebKit.
• Running WebKit's crashtests in Gecko.
• Running WebKit's crashtests in Chromium.

So yeah, I do think they're useful :)

[emilio]

And this was a very simple test runner, there's tons of test-cases that I haven't ran or what not.

[gsnedders]

I'd rather call them crash tests, given that's what they're called everywhere else.

[jgraham]

The reason for not calingthem crashtests was that crashing is only one possible failure mode; they could also fail with an assert or sanitizer failure or similar in vendor infrastructure. But I'm sympathetic to the idea that "load" can be confusing and there is precedent here.

[gsnedders]

I think that's an overly pedantic viewpoint to take when there's prior art in all vendors here already, even if the name is slightly confusing. (Just make all assertions and saniziter failures crashes, pff!)

[jgraham]

https://github.com/web-platform-tests/wpt/pull/20017 is a draft PR for this RFC. It's not done yet (hence draft), but what's there seems to work.

[foolip]

My preference is for -crash.

[jgraham]

OK changed to -crash and crashtests. So I don't think there are more remaining issues and will merge.

[Hexcles]

Doing some archeology here, I wonder when we changed the wait class from wait to test-wait? @jgraham