Safari failing infrastructure tests

gsnedders commented 6 years ago

With Safari TP 63:

/infrastructure/assumptions/html-elements.html
  FAIL Compare CSS div definitions (only valid if pre-reqs pass) - assert_equals: Different value for -webkit-mask-box-image-slice expected "0" but got "0 fill"
http://web-platform.test:8000/infrastructure/assumptions/html-elements.html:71:20
step@http://web-platform.test:8000/resources/testharness.js:1553:30
test@http://web-platform.test:8000/resources/testharness.js:544:22
global code@http://web-platform.test:8000/infrastructure/assumptions/html-elements.html:58:5
  FAIL Compare CSS span definitions (only valid if pre-reqs pass) - assert_equals: Different value for -webkit-mask-box-image-slice expected "0" but got "0 fill"
http://web-platform.test:8000/infrastructure/assumptions/html-elements.html:88:18
step@http://web-platform.test:8000/resources/testharness.js:1553:30
test@http://web-platform.test:8000/resources/testharness.js:544:22
global code@http://web-platform.test:8000/infrastructure/assumptions/html-elements.html:76:5
/infrastructure/server/context.any.sharedworker.html
  FAIL context - ReferenceError: Can't find variable: SharedWorker
global code@http://web-platform.test:8000/infrastructure/server/context.any.sharedworker.html:8:41
/infrastructure/server/context.any.serviceworker.html
  FAIL context - assert_true: expected true got undefined
https://web-platform.test:8443/infrastructure/server/context.any.js:5:18
step@https://web-platform.test:8443/resources/testharness.js:1553:30
test@https://web-platform.test:8443/resources/testharness.js:544:22
global code@https://web-platform.test:8443/infrastructure/server/context.any.js:2:5
importScripts@[native code]
global code@https://web-platform.test:8443/infrastructure/server/context.any.worker.js:8:14
/infrastructure/server/order-of-metas.any.sharedworker.html
  FAIL foo - ReferenceError: Can't find variable: SharedWorker
global code@http://web-platform.test:8000/infrastructure/server/order-of-metas.any.sharedworker.html:9:41
/infrastructure/server/secure-context.https.any.serviceworker.html
  FAIL Use of .https file name flag implies secure context - assert_true: expected true got undefined
https://web-platform.test:8443/infrastructure/server/secure-context.https.any.js:3:14
step@https://web-platform.test:8443/resources/testharness.js:1553:30
test@https://web-platform.test:8443/resources/testharness.js:544:22
global code@https://web-platform.test:8443/infrastructure/server/secure-context.https.any.js:2:5
importScripts@[native code]
global code@https://web-platform.test:8443/infrastructure/server/secure-context.https.any.worker.js:8:14
/infrastructure/server/secure-context.https.any.sharedworker.html
  FAIL secure-context - ReferenceError: Can't find variable: SharedWorker
global code@https://web-platform.test:8443/infrastructure/server/secure-context.https.any.sharedworker.html:8:41
/infrastructure/server/title.any.sharedworker.html
  FAIL foobar - ReferenceError: Can't find variable: SharedWorker
global code@http://web-platform.test:8000/infrastructure/server/title.any.sharedworker.html:8:41
/infrastructure/testdriver/bless.html
  FAIL user activation - promise_test: Unhandled rejection with value: object "NotSupportedError: The operation is not supported."
promiseReactionJob@[native code]

[ ] /infrastructure/assumptions/html-elements.html — https://bugs.webkit.org/show_bug.cgi?id=187052
[ ] /infrastructure/server/context.any.sharedworker.html, /infrastructure/server/order-of-metas.any.sharedworker.html, /infrastructure/server/secure-context.https.any.sharedworker.html, /infrastructure/server/title.any.sharedworker.html — SharedWorker unsupported (https://bugs.webkit.org/show_bug.cgi?id=149850)
[x] /infrastructure/server/context.any.serviceworker.html, /infrastructure/server/secure-context.https.any.serviceworker.html — https://bugs.webkit.org/show_bug.cgi?id=188842
[x] /infrastructure/testdriver/bless.html — #13403

gsnedders commented 6 years ago

Do we want to workaround these bugs, because they aren't per-se Safari running tests incorrectly? Except maybe the bless.html one?

foolip commented 5 years ago

@cdumez @youennf could either of you help triage https://bugs.webkit.org/show_bug.cgi?id=187052?

youennf commented 5 years ago

I cced some folks on bugs.webkit.org. What is the impact of failing these infra tests?

foolip commented 5 years ago

@youennf it's not a big problem, but in order to run infrastructure/ tests in CI (https://github.com/foolip/wpt/pull/5) to ensure that Safari doesn't regress, we'll need to check in expectations about every failure in infrastructure/. That in turn is in service of https://github.com/web-platform-tests/wpt/issues/13299, i.e., preventing regressions of non-infrastructure tests in Safari before they happen.

Understanding why the tests are failing would be great even if it's not fixed soon.

foolip commented 5 years ago

For bless.html, this is the failing test:

promise_test(() => {
  const video = document.createElement('video');
  video.setAttribute('src', '/media/counting.ogv');
  document.body.appendChild(video);
  return test_driver.bless('start video playback', () => video.play())
}, 'user activation');

And the failure is Unhandled rejection with value: object "NotSupportedError: The operation is not supported." That's because of /media/counting.ogv, Safari doesn't support Ogg Theora, so https://github.com/web-platform-tests/wpt/issues/9354.

Switching to /media/counting.mp4 seems to fix the problem.

However, with the same fix, I've also seen flaky timeout or failure due to Unhandled rejection with value: object "AbortError: The operation was aborted." without any extra changes.

Running the tests flips back and forth a lot between the two tabs, which is https://github.com/web-platform-tests/wpt/issues/10398. My best guess is that media playback is being paused or prevented to start in background tabs.

Anyway, this test doesn't need a media resource as well. Browsers that don't allow media playback without a user gesture will leave video.paused as true, so I'll try to change the test.

foolip commented 5 years ago

I've sent https://github.com/web-platform-tests/wpt/pull/13403 to fix bless.html.

gsnedders commented 5 years ago

@youennf probably nothing; https://bugs.webkit.org/show_bug.cgi?id=187052 is the most annoying given it makes it hard to verify behaviour is correct.

foolip commented 5 years ago

The test infrastructure/assumptions/allowed-to-play.html test I added in https://github.com/web-platform-tests/wpt/pull/13622 is also failing in Safari Technology Preview 67.

foolip commented 5 years ago

I've filed https://bugs.webkit.org/show_bug.cgi?id=190775 for that.

foolip commented 5 years ago

Downgrading the priority, this isn't causing any widespread trouble it seems.

gsnedders commented 3 years ago

So current issues:

infrastructure/assumptions/ahem.html since we stopped installing Ahem on macOS
infrastructure/assumptions/allowed-to-play.html due to the inability to automatically change the default autoplay setting (https://bugs.webkit.org/show_bug.cgi?id=190775)
All print reftests due to lack of the WebDriver print command (https://bugs.webkit.org/show_bug.cgi?id=217173)
All sharedworker tests due to its continued disablement (https://bugs.webkit.org/show_bug.cgi?id=218944)
infrastructure/server/http2-websocket.sub.h2.any.js due to lack of support for WebSockets over HTTP/2 (rdar://19558310)
Various infrastructure/testdriver/actions failures (rdar://75494624)

With that being the status now, I don't know if it makes sense to keep the meta issue open?

web-platform-tests / wpt

Safari failing infrastructure tests #12621