WKWebView malfunctions in legacyScreenSaver in Sonoma

[xmddmx]

In all of the macOS Sonoma betas so far, any screensaver using WebKit will not work properly.

The main issues we've found so far are:

• video does not play
• animations that use RequestAnimationFrame do not work.
• attempts to work around RequestAnimationFrame bugs using SetInterval() also do not work (setInterval only fires once per second).

This is not specific to WebViewScreensaver - it seems to affect any third party screensaver that uses WebKit.

Please also see https://github.com/JohnCoates/Aerial/issues/1305 which has a long list of Sonoma screensaver bugs (many of which we have found workarounds for).

This has been reported to Apple as FB13094564, and I also submitted a TSI about this today (since it's still broken in today's Sonoma RC release).

[dopeytree]

No settings options in Sonoma

[markusbkk]

• animations that use RequestAnimationFrame do not work.

This is a threading issue. For whatever reason Apple has decided to aggressively optimize CPU utilization.

This old Stackoverflow gave me the right idea and made me investigate in that direction.

My own PoC went from a still image to a somewhat bearable (but still unstable over time) 20 FPS after introducing web workers to it. It's late at night and I need to get up early for my day job, but I'll keep y'all posted as I further investigate and optimize things.

[jonathonw]

• animations that use RequestAnimationFrame do not work.

This is a threading issue. For whatever reason Apple has decided to aggressively optimize CPU utilization.

This old Stackoverflow gave me the right idea and made me investigate in that direction.

That's at least part of the story here-- document.hidden === true in a screensaver in Sonoma, so it looks like it's preventing/throttling animations because it thinks it's not visible and in the background. I don't see any immediately obvious way to override that in WKWebView either, although I might be missing something.

Still broken in the released version of Sonoma 14.2, by the way.

[markusbkk]

• animations that use RequestAnimationFrame do not work.

This is a threading issue. For whatever reason Apple has decided to aggressively optimize CPU utilization. This old Stackoverflow gave me the right idea and made me investigate in that direction.

That's at least part of the story here-- document.hidden === true in a screensaver in Sonoma, so it looks like it's preventing/throttling animations because it thinks it's not visible and in the background. I don't see any immediately obvious way to override that in WKWebView either, although I might be missing something.

Still broken in the released version of Sonoma 14.2, by the way.

Yea. I didn't know how to put it into words but I figured that, while it was rendering in the front to the human eye, it wasn't really rendering in the front to the system (it's 2 AM over here, so I'm having a wee bit of trouble choosing my words).

Out of curiosity. Where did you find the document.hidden === true? I'm new to macOS screensaver development so I have yet to master the proper way to debug those.

[jonathonw]

I got document.hidden from within the web view (added JavaScript code to show it to a page that WebViewScreenSaver was set to display)-- it's the DOM property that tells you if the page is visible or not. See https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API

[markusbkk]

I got document.hidden from within the web view (added JavaScript code to show it to a page that WebViewScreenSaver was set to display)-- it's the DOM property that tells you if the page is visible or not. See https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API

Ah. Ok. Thank you. I was wondering if I maybe missed something and there were graphical debugging facilities for screensavers under macOS.

[xmddmx]

Here is an outline to the solutions I've found:

1. Add in a polyfill for RequestAnimationFrame, SetInterval, and SetTimeout which uses WebWorkers to trigger the callbacks. Although the main Javascript thread is throttled, it turns out that Worker threads are not, so you can get (approximately) the right timing. The Worker thread raises an event back on the main thread.
2. Video playback can be fixed by grabbing an animation frame from the video as it plays. No idea why this works, but it does. Create a 2D context, and callctx.drawImage(video,0,0,canvas.clientWidth, canvas.clientHeight). You only need to do this once.
3. Sometimes, the OS will still put your screen "to sleep" and throttle it further. I've seen this on multimonitor systems on monitor 2, 3... A workaround is to play a silent M4A <audio> which tricks the OS into keeping that screen awake.

[markusbkk]

Here is an outline to the solutions I've found:

1. Add in a polyfill for RequestAnimationFrame, SetInterval, and SetTimeout which uses WebWorkers to trigger the callbacks. Although the main Javascript thread is throttled, it turns out that Worker threads are not, so you can get (approximately) the right timing. The Worker thread raises an event back on the main thread.
2. Video playback can be fixed by grabbing an animation frame from the video as it plays. No idea why this works, but it does. Create a 2D context, and callctx.drawImage(video,0,0,canvas.clientWidth, canvas.clientHeight). You only need to do this once.
3. Sometimes, the OS will still put your screen "to sleep" and throttle it further. I've seen this on multimonitor systems on monitor 2, 3... A workaround is to play a silent M4A <audio> which tricks the OS into keeping that screen awake.

So if I understand you correctly, you're saying to use the worker threads solely to trigger the next tick while still running all canvas operations in the main thread? My own experiments with OffscreenCanvas produced markedly worse results (only about 1/3 of the performance compared to the preview or running it inside a browser).

Will definitely have to give 3. a try.

[xmddmx]

Yes, do everything on the main thread, use Workers only to trigger the events.

One final issue:

4. CSS Animations don't work. The only workaround I've found for this is to rewrite my JavaScript to use custom Animators for each property. This is only practical if (A) you only have a few different kinds of animations and (b) you control the Javascript. In my (commercial, closed source) project this is posisble, but if you are trying to run an arbitrary web page, I can't see how to make this work.

[markusbkk]

Yes, do everything on the main thread, use Workers only to trigger the events.

One final issue: 4. CSS Animations don't work. The only workaround I've found for this is to rewrite my JavaScript to use custom Animators for each property. This is only practical if (A) you only have a few different kinds of animations and (b) you control the Javascript. In my (commercial, closed source) project this is posisble, but if you are trying to run an arbitrary web page, I can't see how to make this work.

Was about to ask about CSS animations. Never tried them inside any screensaver so I wasn't sure if there was an issue to begin with. Thanks for the heads-up on that.

[pbi-qfs]

For the famous Flying Toasters Screensaver, there is a popular CSS only implementation: https://www.bryanbraun.com/after-dark-css/all/flying-toasters.html (from https://github.com/bryanbraun/after-dark-css) - As wrapped screensavers, they are all dead on Sonoma now :(

[markusbkk]

@xmddmx

Any idea what I'm doing wrong here? I'm trying to follow your advice RE using a webworker solely to trigger the next repaint.

window.addEventListener('DOMContentLoaded', () => {

    const workerScript = `
      onmessage = (e) => {
        postMessage("callback");
    }
    `

    const blob = new Blob([workerScript], { type: 'text/javascipt' });
    const worker = new Worker(window.URL.createObjectURL(blob));

    let angle = 0;
    const ctx = canvas.getContext('2d');

    function drawHeart() {
      ctx.fillStyle = 'blue';
      ctx.beginPath();
      ctx.moveTo(125, 50);
      ctx.bezierCurveTo(75, 0, 0, 75, 125, 175);
      ctx.bezierCurveTo(250, 75, 175, 0, 125, 50);
      ctx.fill();
    }

    function drawAndRotate() {
      ctx.clearRect(0, 0, canvas.width, canvas.height);
      ctx.save();
      ctx.translate(125, 125);
      ctx.rotate(angle);
      ctx.translate(-125, -95);
      drawHeart();
      ctx.restore();
      angle += Math.PI / 180;
      worker.postMessage("draw");
    }

    drawAndRotate();

    worker.onmessage = (e) => {
      requestAnimationFrame(drawAndRotate);
    }
    })

[xmddmx]

Apologies, I was never able to get requestAnimationFrame to work, so I ended up just doing setInterval(..., 16) inside the worker instead to get approximately 60 fps.

[xmddmx]

Also, I think you have it backwards - the setInterval() call needs to happen inside the worker script, and when that event triggers, it should call the postMessage() function.

Something like this:

const workerScript = `
      onmessage = (e) => {
        // inside the worker's thread, we do the setInterval call
        setInterval( function() { postMessage("setInterval callback") }, 16  );
    }
    `
  worker.onmessage = (e) => {
      // do your animation here in the main thread
    drawAndRotate()
    }

[agologan]

Think I got everything in v2.3 but feel free to comment if you still encounter issues, or better yet open a new issue if it seems unrelated.

Btw, your initial comments were very helpful in figuring this out. Thank you 🙇