Open JeffreyBlanz opened 3 months ago
Thank you for doing this due diligence.
I offered to do this proposal too -- but now that you've started this (and reused some of my writings too -- make sure you list me as a coauthor). I'll get behind this effort too!
Any assistance I can offer in furthering this proposal, let me know.
Thanks @JeffreyBlanz. Would you mind generalizing the proposal? Talking about "Apple's sales" etc. is probably not appropriate so it would be good to drop that. As you noted, this is not OS dependent and the mode could switch based on environmental conditions (e.g., the user could plug their device into a power source).
For the proposal itself, I think it's presumptuous to ask for permission. As with Apple's Game Mode, the user has the ability to turn it off without needing to ask for permission: it's something that happens automatically upon request. Further, the refresh rate can change for whatever reason, so staying at the high frame rate is never guaranteed.
As such, it might be good to look at making this an option when requesting fullscreen. As such, I'd suggest looking at FullscreenOptions
and seeing how this could integrate into that.
That would allow the user agent to opt into the higher modes whenever requested, while leaving the user and the user agent to decide if/when it's enabled without exposing if the hardware supports such as mode.
As you pointed out, this is also not about games... this has a lot of use cases beyond gaming, so a generalized opt in for high refresh rate would be appropriate.
Given the above, can you refactor the proposal to take the above into consideration?
Since some here are not familiar with me, I would like to introduce myself better. I am the creator of Blur Busters and a popular motion testing website, testufo.com and HDR version at beta.testufo.com.
I'm in over 35 research papers (See Google Scholar), and the best learning point about me is to read Blur Busters Area 51 Display Science, Research & Engineering.
TestUFO, at 8 figure count of unique visitors per year, is now the world's most popular high-framerate-educating website. It has many tests for teaching, for demos, for scientific research, etc. Some TestUFO's only require human eyes, and some require equipment (camera, high speed camera, oscilloscope).
More interesting animation examples are Frame Rates Demo, Game Panning Demo, Black Frame Insertion Demo, Ghosting Test for Cameras, Local Dimming Test, Visual MPRT Test, Stutters-to-Blur Continuum Demo, and dozens others.
I have been noticing that 120Hz is becoming gradually more pro-mainstream, and TestUFO has become more popular for users to see for themselves the benefits of high refresh rates. TestUFO was made 10 years ago when 120 Hz was still a much more luxury niche.
Some motion education pages are only properly educational at certain refresh rate and frame rate minimums, for certain physics & science reasons (e.g. Comparing two frame rates that are BOTH well above flicker fusion thresholds, e.g. 120fps versus 240fps).
There are mudane physics reasons why I want the world's most popular Hz/framerate "education-in-a-link" to properly teach a user with minimum effort.
Besides, no permission request is needed for Chrome, Edge, Safari, FireFox. Safari is the lone holdout, and the permission request is optional; only needed if not defaulting to uncapped.
Unfortunately, I have to add this section because a lot of developers are unfamiliar wit the ergonomic benefits of high-Hz, even outside gaming.
The ergonomic benefit is much more dramatic on the 60-vs-120 OLED (2.0x human visibiility difference) than LCD (sometimes only 1.1x difference due to slow GtG). And even more dramatic for 120Hz vs 480Hz OLED, than 60Hz vs 120Hz due to the 4x geometric. The diminishing returns goes a long way, but it's based on geometric and not a cutoff.
From moving map test www.testufo.com/map is this amount of motion blur, depending on which refresh rate you are at:
But static images can only teach so much. I need to teach it en motion, and some educational motions are only properly educational at "Hz > X" where X may be a triple digit number.
...I just find an easy way for me to teach high-framerate-mandatory concepts to the millions of existing 120Hz Apple displays in an easy link click. How? It's impossible without fiddling an Advanced developer setting on many of them.
Thanks @JeffreyBlanz. Would you mind generalizing the proposal?
I volunteer to do it within this month!
For the proposal itself, I think it's presumptuous to ask for permission.
Other browsers don't need to ask for permission (only specific browsers like Safari that choose to cap at 60fps). And you see the additional mainstream use cases beyond gaming.
Apple is a privacy sensitive company. Plus, other display-related APIs ask for permission (Windows Management API) and this proposal could be rolled into that in theory (recycle the same permission).
As such, it might be good to look at making this an option when requesting fullscreen.
It's not exclusive to fullscreen or gaming. 240-480Hz is good for office productivity and ergonomics too (scrolling, panning). My MacBook M1 scrolls fine on my 480Hz and 240Hz monitor, and looks miraculously CRT clear.
120 versus 480 at GtG=0 (OLED) has a bigger ergonomic scrolling difference than between 60Hz and 120Hz.
TechSpot and other mainstream reviewers even said 240Hz was much better than expected for productivity (scrolling that looked better than even an OLED 120Hz iPad. 480Hz and 1000Hz, ultimately not for gaming anymore (in ten years), much like 4K is no longer a luxury product.
That would allow the user agent to opt into the higher modes whenever requested, while leaving the user and the user agent to decide if/when it's enabled without exposing if the hardware supports such as mode.
GENIE ALREADY OUT OF BOTTLE ALARM
All major browsers (Chrome, FireFox, Edge, etc) do it on all platforms (Mac, Windows, Linux Wayland, Android), except for Safari.
This ended up being kind of a defacto industry standard. It simply offers a simple solution for Apple & Safari that satisfies Apple's concern about privacy & battery (but can be open to any browser vendor).
Imagine, a link that does a 5-second teaching demo, whether to educate about display physics, or simply to sell the benefits of high Hz. Why should users have to be instructed to go through a Developer menu for this?
REASON:
Translation friendliness: I now think about this whenever I write tech specs, like during my git commit to W3C HTML 5.2 in year 2017 (section 7.1.4.2 at the time, although it's renumbered now). I try to be cognizant that various spinoffs of Blur Busters content is read by over 150+ countries, with over 100M people worldwide at least briefly seeing my UFO logo in the tests displayed by my content creators and/or links to training materials that buttress research/measurements (same motion test website is used for both).
Kudos on the MacBook M1, my main TestUFO development machine, in Visual Studio Code now. Watching for OLED MacBooks, hopefully 240Hz+
If you want to read more:
I love the battery life of my 120 Hz MacBook M1 during TestUFO development. A 120Hz MacBook is actually now my main portable development machine for TestUFO, although it's not the best 120Hz display (Being LCD rather than OLED, it's not as human visible 60-vs-120 difference as on the OLED Apple iPad). I look forward to upgrading to an OLED MacBook, and also look forward to the time when Apple introduces 240 Hz OLEDs in at least one product in two years from rumors I hear.
Also, 120Hz Apple LCD is pretty slow compared to a 120Hz desktop gaming monitor, since power efficiency also slowed the LCD pixel response (lack of overdrive algorithm). That's worth it for a 10-hour MacBook battery life, and I understand Apple's rationale. That being said, slowing down the pixel response diminished 60-vs-120 human visibility differences, but on the new 120Hz OLED iPad, running iOS 18 with 120fps, TestUFO 60-vs-120 looks simply stunning amazing. Try www.testufo.com/framerates-versus on the OLED iPad Pro that is manually uncapped (using the setting and the beta iOS 18). Instead of 60-vs-120 being a mere 1.1x human visibility difference, it's now suddenly a 2.0x human visibility difference during fast smooth scrolling/panning situations.
There are some debates between me and @smfr of Apple in the past. But perhaps talk to that person in the displays department who has UFO sticker on their MacBook or iPhone instead (there's a few), as the UFO is like a university graduation symbol to them in the temporal sphere. They can show how 120Hz is crap or good.
Plus also for competitor-research, perhaps witness a Windows running framerate=Hz on a 480Hz OLED (even on an Intel internal GPU) more smoothly than MacBook running on a 120Hz LCD, since prioritization has been on power management over stutter-free TestUFO. But easy activation of a framerate uncap would be nice, even if behind a Game Mode.
But the new M1-M4 chips are quite amazingly performant, and my MacBook even runs fine on an external 240Hz monitor now too (Safari now can be forced to do 240fps, thankfully, at least with the laptop lid closed and only external OLED active).
Some people inside Apple formerly thought there wasn't performance on Macs to run at 480Hz. This is false. There is framepacing jitter because sometimes executions are coalesced at less precise times to maximize sleeping between refresh cycles, which can jitter a bit, because sometimes the ideal-power requires a bit of aberration from executing at refresh cycle timestamps to merge all the CPU/GPU workloads that needs to be executed in a surge. It's a common engineering power management optimization, which may have partially helped (in addition to slow 120Hz LCDs) make some Apple employees blind to the benefits of 120Hz+. So TestUFO does understandably stutter more on MacBooks than on Windows. However, with the M4, it's much more reliable, and perhaps more testing is needed.
Thanks to OLED, we can have "great battery life" + "great color" + "fast pixel response", with more organic 120fps 120Hz improvements visible to humankind. Also, I want to point out TechSpot about why Apple should add 240Hz to their OLEDs:
TechSpot wrote as part of their OLED burnin-test review:
"The 240Hz refresh rate at 4K is much better than I was expecting for productivity work. Relative to the 144Hz LCD I was using, the combination of a higher refresh rate and faster response times makes this QD-OLED much nicer to use for everyday tasks. Especially when browsing the web and scrolling through text, the speed and refresh rate combination is noticeable and provides a smoother, clearer experience – so it's not just gaming where you'll benefit from a 240Hz refresh rate."
4x geometrics produce a much more stunning wow effect (60-vs-240) than (60-vs-120), which can help sell Apple product. When I show off 60-vs-120 on an OLED (especially bigger screens like OLED tablets, OLED laptops, and OLED desktops), people suddenly want to buy a 120Hz product. So yes, it helps sell. Much more visibly than a 120Hz LCD Apple product. You want to VHS-vs-8K it, not 720p-vs-1080p it. Big jumps + GtG=0 is key.
Now it's possible at much lower low power levels thanks to OLED and the M-series, and the latest iOS 18 beta version of TestUFO uncapped to 120fps looks absolutely gorgeous on the 120Hz OLED iPad Pro. Massively more visible than on my 120Hz LCD MacBook M1 that's my TestUFO development machine.
But I had to jump through a lot of hoops to do that (install iOS 18 + also change a setting), so it's not an easy successful "5 second educate in a link" to default unconfigured 120Hz Windows systems.
However, it's become clear (on Windows side) that 60-vs-240 sells even more powerfully, as a 4x improvement (like 1/60sec vs 1/240sec camera shutter). OLED uncaps the humankind benefits of refresh rates to the mainstream. Especially with the emerging anecdotes of 240Hz OLED ergonomics for productivity for non-gaming use cases. But a baby step of easy 60-vs-120 comparisions in a browser on an iPad people can spontaneously show off to others. Today, that's not easy with the mandatory need to fiddle with settings (at least there's now that -- formerly, it was impossible with a hard 60fps cap that formerly couldn't be overriden at all).
As an outside smart analyst (and Apple is notoriously secretive), I know other companies though. It requires coordination between many departments:
So, an orchestra between multiple departments regardless of platform. Apple controls all of them internally and I have no idea how Apple does it, but I know there's been suggestive public hints over the last 10 years of 120Hz disagreements (very pro versus very anti). But bottom line, M1-M4 framepaced TestUFO fine at 480fps 480Hz on an external monitor in my tests (once the OS compositing power-optimizing jitter was solved -- not Safari's fault!). The nice thing is that OLED 60-vs-120 quited a lot of the dissenting voices... So, the weak link, Apple employees, was actually in a different Apple department!!
The operating system compositing (in my microsecond-accurate analysis), has intentional jitter that sometimes accidentally adds stutters to TestUFO despite using only 10% of an Apple GPU, due to all the execution-coalescing and microsleeps-between-refreshcycles. So it's potentially just a timing issue, not a performance issue. However, granting Game Mode API, means you could switch to a more performance mode that also includes more accurate graphics compositing timings (When prodded as such with hacks, M4 easily did 480fps 480Hz, in my tests on an external OLED).
Yes, generic industry proposals must avoid these kind of details. For this proposal I will remove the apple-sell portion, but I had to explain why I added that to the earlier proposal (that Jeff Blanz did).
Yes, there's a 10-year history between me and various Apple employees (on public venues like these) trying to advocate their framerate uncapping, and it's a long slog.
It's a breakthrough to finally see Apple add a flag override ("Prefer Page Updates Near 60fps" that can be disabled to uncap frame rates). But that's still not easy enough for a quick way to market 60-vs-120 to end users to help sell the benefits of mainstream non-gaming use cases of 120fps and 240fps.
Just explaining to add more detail why it slipped into this venue from a paragraph copy-and-pasted from somewhere else I posted. I actually caused some people to buy 120Hz products (including Apple products) showing them 60-vs-120 on an OLED iPad, so, there is a genuine, actual "sell" factor, that portions of Apple may have apparently overlooked.
Earlier, the power concerns and pixel response limitations meant power management outweighed the ability to sell 60-vs-120. But with the new OLED 120Hz iPad Pro, it has fantastic battery life while still doing 120fps 120Hz, while still being much more human-visible than previous LCDs. Now the sell-factor outweighs the power-management-concern factor of Safari having a permanent 60fps cap out-of-box by default.
It's time for Apple to internally re-evaluate why they're not uncapping Safari by default (like everyone else now), but I just wanted to offer an olive branch of an alternative quick mechansim (Permission request for uncapping www.testufo.com framerates without me displaying instructions) that allows Apple to cap by default, but makes it easy for webpages to uncap in less than 2 seconds (one-time permission request). I am open to ideas of alternatives that Apple may prefer, while helping sell 120Hz product. Remember, I am a big-time Apple customer, and I am communicating how I've sometimes sold 120Hz to other Apple users.
All fixable, and voila -- 480fps 480Hz TestUFO works fine on some Apple M-series with zero TestUFO stutter! (But lots of manual configuring, unlike automatigically works on Windows). You do have to purchase a third party 480Hz OLED, and close the laptop lid (to prevent a 60Hz-120Hz clock interference), and do a bunch of power management disable tweaks (something that automatically happens when you are playing video games, anyway).
The interdepartmental orchestra inside Apple, not visible to public, helped keep Safari keep a permanent 60fps cap for much longer than competitors. It's justified -- my MacBook is my main TestUFO development machine, and can do 10 hours at a cafe just fine. Wonderful battery life, despite me overriding Safari to 120fps 120Hz during TestUFO development.
I want a "easy sell 60-vs-120 in a single web link" mechanism, Apple, that I can tell anybody to do... (without asking them to surf through the Advanced settings).
Closeout: Yes, I'll remove Apple references from the proposal. But you can clearly see where I'm coming from; this proposal is also trying to satisfy a lone holdout's privacy & power management needs.
I'm partially rewriting because I hate the Game Mode focus, actually. I want to delete the Introduction and replace it with something broader to better introduce:
For a long time, multiple web browsers had frame rate caps for various reasons, such as simplicity (e.g. 60Hz timer event) or power management (e.g. battery consumption) or arbitrary (e.g. decision made by a single developer or team). These frame rate caps can be less than the display refresh rate, which can affect user experience.
This specification aims to standardize a simplified method that websites can use to access higher frame rates for many use cases, in situations where the browser frame rate is capped by the browser engine. Today, only a few browsers still have frame rate caps outside of low-battery mode.
Before around 2012, some browsers ran off an internal 60 Hz timer event, but that changed. Around this time, both Chrome and FireFox uncapped their frame rates, and Mark of Blur Busters did 120 Hz Browser Tests in year 2013 and discovered that most web browsers had uncapped frame rates that worked fine on early 120 Hz gaming monitors over a decade ago.
At the time, Internet Explorer and Safari became the holdouts. Internet Explorer had an internal 105 fps frame rate cap that automatically halved frame rates if the refresh rate was above 105 Hz. This was established all the way back to the CRT days when CRT refresh rates were generally only up to about 85 Hz to 100 Hz, and the number was just above the 100 Hz CRT number.
Now that Internet Explorer is replaced by Edge, Internet Explorer is no longer relevant to this discussion. All Chromium browsers have the same uncapped framerate behavior - Chrome, Brave, Vivaldi, etc. These browsers have been successfully tested to 540 Hz on commercial displays, and tested to 1000 Hz on laboratory displays (Windows 10 supports up to 500 Hz, and Windows 11 supports up to 1000 Hz).
Since 120Hz is much more common and premium-mainstream than it was in 2012, this has become a more pressing issue than back then. Originally, high native refresh rates were only on specialized gaming displays of lower quality (e.g. TN panels), while now high refresh rates have become inclusions on more mainstream and higher quality displays (e.g. HDR OLED panels) than back then.
Today, this is the state of frame rate capping:
This proposal aims to find a way to make it easier for the user to uncap their web browser for any purpose, within the OS currently allowed frame rates and refresh rates.
Hybrid approaches are possible, e.g. Game Mode automatically grants permission to uncap frame rate, while you could need a different API to uncap frame rate outside of Game Mode. Browsers that already uncap by default (e.g. Chromium) can automatically grant permission. Discussion is welcome.
@mdrejhon, appreciate the updates. Can I kindly ask to try to keep replies succinct 🙏 (try to limit to 500 words max). Otherwise this issue will become unwieldy and difficult to follow. If you find that you want to add background details, please use <details>...lots of details here...</details>
. Like this:
About game mode, it's not exclusively about uncapping frame rates (which may end up breaking some games that are expecting to run at ~60hz), it also impacts CPU/GPU prioritization, etc.
<details>
As a person harkening back to 1990s era Internet, and born deaf, I write big walls of text. I'm born deaf, and I'm a very visual researcher in the display-temporal domain (jitter, lag, GtG, MPRT, Hz, VRR, framerate, etc).
About game mode, it's not exclusively about uncapping frame rates (which may end up breaking some games that are expecting to run at ~60hz), it also impacts CPU/GPU prioritization, etc.
Yes, that's exactly why I suggested a permission API.
It means the application is explicitly above-60fps aware, and needs access. The general good news is that most browsers have been generally uncapped on all platforms (Windows, Android) except for Safari and Apple, so most games have removed the 60fps expectation. But I extend privacy-friendly options here.
My alternative is to teach inexperienced visitors how to change a developer flag. I already warn all 60fps Apple users with a message.
Update, I was experimenting the easiest way to have users reliably demo high-framerate TestUFO in Mac Safari. With these steps, 240fps 240Hz and 480fps 480Hz Safari works jitter-free in TestUFO on MacBook Pro M1, the very first M1. (cc:@smfr) It works now [PHOTO], but it's still convoluted, but this is the simplest instructions:
These instructions Apple Safari/Chrome surprisingly does concurrent multi-Hz support better than Microsoft's Edge/Chrome does. Photos of 120fps-120Hz TestUFO and 240fps-240Hz TestUFO running concurrently on MacBook M1:
@JeffreyBlanz and @marcoscaceres any comments before I continue editing? Also, beta.testufo.com is finally public to the whole world, if you want to test high Hz+HDR simultaneously.
Apple is a very privacy-sensitive company, so if Apple does not want sites to know the refresh rate of their display, it can be put as part of a permission request, or rolled into an existing one like Display Management API. This could be a compromise that @smfr likes.
I already pop up a small warning to all 60fps Apple users at beta.testufo.com, and will be expanding that to display a detailed step by step guide to users on how to disable the 60fps flag. Users can choose to do so, but I'd like a more proper way than inexperienced users fiddling a dev flag.
Intent of this proposal is to uncap everything at the browser compositor level all the way to the OS-allowed limit: HFR video (120fps realtime), scrolling, animations, 2D, WebGL, CANVAS, Game APIs, Canvas 2D, requestAnimationFrame, and any web animation related APIs.
All of that already works fine >60fps in all current major browsers (Safari, FireFox, Edge, Chrome, etc) on all major platforms (PC, Mac, Linux Wayland, Android, iPhone iOS 18+, iPad, 120Hz PlayStation, 120Hz XBox, 90Hz+ VR browsers, etc). All prior art, all defacto standardized.
It's merely simply a hoop-jumping manoeuvre for Apple Safari users where I have to teach inexperienced users how to fiddle a developer flag (Prefer Page Rendering Updates Near 60fps).
Doing >60fps already works in all HTML5.2+ APIs in all major browsers on all platforms. The last holdout, 120Hz iPhones does 120fps TestUFO as of iOS 18+ via byzantine flag change.
This WICG proposal is simply to find a privacy-friendly opt-in method uncap single variable flag (LimitFramerateTo=60).
FireFox has a framerate cap setting too (turned off by default). Other browsers can decide to enable it by default, and use this WICG proposal to uncap. So this is not Apple-exclusive.
This is simple. There are no other dependencies. All HTML5 features already work fine at 60fps, 90fps, 120fps, 240fps in all major browsers on all major platforms, as of year 2024. We do not need to committee-overcomplicate what already works.
Sites that specifically request uncap (by any mechanism) means they're >60fps-aware.
@mdrejhon I think the rewrite above is good and you should continue.
I would like to know more about why @marcoscaceres thinks it might not need to be a permission prompt. More specifically; what are Apple's views on privacy with proposal, and what is Apple's stance on the "prefer page rendering updates near 60 fps" feature flag being default off instead of on?
Assuming this is a privacy concern and the feature flag is staying default on, why would a permission prompt not be ideal?
Ideally, I'd love Apple to a mostly-unconditional API call to uncap upon a click (like Pointer Capture or Fullscreen requires). It doesn't have to be permission-based, as long as it's an easy global-all-APIs uncap for the specific site requesting it (not other sites).
But indeed want to accomodate privacy concerns. Open to ideas for global uncap!
Introduction
On iOS, iPadOS, MacOS, and Android there is a game mode that can be enabled with fullscreen apps that are designated as games. There is currently no way to enable the system game mode in a PWA or normal web engine. On iOS, iPadOS, and MacOS users also have to manually turn off "Prefer Page Rendering Updates Near 60fps" in Developer Feature Flags. This feature flag also does not apply to PWAs, meaning PWAs are stuck at 60fps (see webkit bug 272226). This API intends to add the system Game Mode support as well as enabling native screen refresh rates through a permission prompt for web apps and browsers. Native screen refresh rate can be enabled independently from game mode as well.
Why High Frame Rate is Important Outside of Gaming
120fps was less visible on LCD so this a low priority, due to poor performance of LCD. However, with the dramatic visibility increase of 60-vs-120 on OLED displays (even TechSpot discovered 240Hz OLEDs was much better than expected for office productivity). Even 120Hz-vs-480Hz on OLED was recently scientifically discovered to be more human-visible than 60-vs-120 on LCD, due to a geometrics-style effect (4x vs 2x) during fast motion.
Use Cases
Solution: Game Mode Permission API
In this API there will be two types of permission prompts. One for just native screen refresh rate, and one for enabling the system game mode as well as the native screen refresh rate.
When a website asks for Game Mode permission, the browser will popup a message asking the user for permission to use Game Mode. Upon approval the browser will enable the system game mode and disable any limitations on screen refresh rate.
When a website asks for High Refresh Rate permission, the browser will popup a message asking the user for permission to use High Refresh Rate. Upon approval the browser will disable any limitations on screen refresh rate. CPU and GPU prioritization will not be enabled.
Note: Both permission prompts will only work if it's a same-origin or permitted-origin (so ad banners can never request it).
Solution Sequence
Examples
A possible permission-request message could be: "Webpage is requesting permission to enable Game Mode. This may consume more battery power. Allow? Yes/No"
Another possible permission-request message could be: "Webpage is requesting permission to enable High Refresh Rate. This may consume more battery power. Allow? Yes/No"
Privacy & Security Considerations
Note: By default, webpage MAY NOT discover max Hz or if Game Mode API is enabled if the page is currently capped (e.g. Limit to 60fps), this may resolve privacy concerns if a company is concerned about webpages discovering refresh rates of visitors. However, implementors MAY provide a separate discovery mechanism for refresh rate, much like existing discovery mechanisms for screen resolutions. Another possible Implementation would be for a webpage to return is "60 Hz" versus "Above 60 Hz", via some discovery mechanism whether or not the browser is framerate capped or uncapped. Even if it's simply detecting it simply via a non-immediate-fail on a Game Mode Permission Request.
References
Coauthor: @mdrejhon (Creator of TestUFO)
mdrejhon Similar Bugzilla Proposal: https://bugs.webkit.org/show_bug.cgi?id=274372
Apple: https://support.apple.com/en-us/105118
Android: https://developer.android.com/games/optimize/adpf/gamemode/gamemode-api
PWA Bug: https://bugs.webkit.org/show_bug.cgi?id=274372