Better support for non-60hz display rates

Goal

Support non-60hz display rates. This includes:

the 50hz standard used in Europe, Australia, New Zealand, and parts of Africa, Asia, and South America.
high refresh rate displays (many Androids and gaming monitors)
ProMotion variable refresh rate displays (latest iPhone models)

I want p5play developers to be able to share their games with the whole world and have it run well on any capable device.

p5play game developers and players shouldn't have to worry about frame rates.

The Problem

p5.js sets the default _targetFrameRate to 60. It uses requestAnimationFrame but just avoids drawing anything at a higher rate than 60fps. So users with higher refresh rate displays are stuck at 60hz. But if the user's display rate is lower, like 50hz... umm Houston we got a problem! The physics simulation will run 16% slower than real time because by default, p5play updates the physics simulation by 1/60th of a second at the end of each draw call.

By default q5.js will run the draw loop using requestAnimationFrame based on the user's display rate. This would also make the physics simulation too slow on 50hz displays and way too fast on high refresh rate displays.

Personally, as a tech consumer, I've been a big fan of increasing visual fidelity over smoothness, opting for 4K over higher refresh rates. For me the difference between 4K and 1080p is HUGE, but with refresh rates higher than 60hz, I can't really tell the difference. Maybe my eyes are slow or something lol. All the devices I personally own can "only" do 60hz. So my personal biases led me to not consider this major problem until now.

Also it'd be nice if p5play could limit the physics simulation to 30hz if the user's device isn't capable of achieving 60fps.

I was quite conflicted on how to approach this problem, so I did research on Unity.

How Unity does it

Unity separates the game's physics updates from the frame rendering.

https://docs.unity.cn/520/Documentation/Manual/TimeFrameManagement.html#:~:text=Unlike%20the%20main%20frame%20update,the%20last%20physics%20update%20ended.

Unity uses a fixed timestep of 1/50th of a second for physics calculations to ensure consistent physics simulation, regardless of the frame rate.

Unity uses a variable timestep for rendering and general game logic, which is handled in the Update method. This method is called once per frame, so the frequency can vary depending on the display rate.

In between physics updates, Unity interpolates the positions of physics objects for rendering. This means that even if a physics update hasn't occurred for a particular frame, Unity will estimate the current position of the object based on its previous and next calculated positions. This allows the object to appear to move smoothly, even though its actual position is only being updated at the fixed physics timestep.

This approach allows Unity to provide consistent physics simulation while still rendering as smoothly as possible based on the performance of the device.

I think implementing something like this in p5play would be ideal, but how?

Frames

If the goal is to not require developers or players to worry about frame rates, that means not having a default frame rate. That means maybe frames shouldn't be used as a unit of time measurement anywhere in a game's code. Yikes!

The collision handling system stores how long players have been colliding in frames.
Input devices store how long the user has been doing an input action in frames.
Animations use the ani.frameDelay property to define how many frames an image in the animation is displayed before the next image is displayed.

The case for abstracting frames

Switching from frames to seconds would require making tons of breaking changes to p5play.

Also p5.js primarily uses frameCount as its measure of time.

@davepagurek suggested an alternative solution:

I guess there's maybe a world where "frames" are like css pixels, and one "frame" as a unit might actually represent multiple real frames under the hood on high refresh rate displays? not sure if that's more or less confusing though haha like if everyone defined sketches as if it were 60fps, but sometimes the frame count is a fraction instead of a whole number

More info: https://www.quirksmode.org/blog/archives/2010/04/a_pixel_is_not.html

I do find using frames to be more convenient than milliseconds, a level of precision that's not required for typical user input handling.

I also agree that this issue with higher refresh rates is a bit analogous to the challenge that high pixel density displays posed to developers over a decade ago.

When Apple first introduced high pixel density displays to consumers, I thought the abstract re-branding of pixels was confusing, now it seems perfectly natural. Yet, was that only acceptable because web developers no longer needed to care about real pixels? With retina displays users could zoom in and out without really compromising the visual appearance of text, which Apple had just a few years prior been boasting about always displaying pixel perfect on lower resolution displays.

Have we gotten to that same point with high refresh rate displays above 60fps? Perhaps so.

Unfortunately, p5.js just released v1.9.1 (EDIT: and later v1.9.2 as well) without the necessary bug fixes that p5play v4 will rely on.

EDIT: p5.js released v1.10.0 in July 2024 with a fix to deltaTime

iOS Safari's default limit is 60fps

I did some testing on an iPhone 13 and to my surprise found that p5play was running at 60fps.

I learned that by default even on iOS devices with ProMotion displays, Safari and WKWebViews are limited to 60fps, assumedly to save battery. This can be changed in settings, although I wouldn't expect that many people do it. Low power mode can also limit some iPhone's display rate to 30hz to save battery.

So it turns out existing p5play v3 projects should just run fine on any current iPhone.

Frames

Since frames are small integers, users can use equivalence checks == in their code instead of doing range checks on inconsistent decimal values.

See this example that checks if the user has been pressing the mouse for 10 frames.

if (mouse.pressing() == 10) {
  // run some code one time
}

How could an equivalent check be written if the time was stored in seconds?

if (mouse.pressing() == 0.16666666666) {
  // run some code one time
}

AHHH! It's awful. Also it wouldn't even work with different refresh rates like PAL 50fps. 8 frames would be 0.16 seconds and 9 frames would be 0.18.

ChatGPT says in Unity that developers would have to write code like this:

float mousePressedTime = 0f;
bool codeExecuted = false;

void Update() {
    if (mouse.pressing()) {
        mousePressedTime += Time.deltaTime;
        if (!codeExecuted && mousePressedTime >= 0.166f) {
            // run some code one time
            codeExecuted = true; // Set the flag to true after executing the code
        }
    }
    else {
        mousePressedTime = 0f;
        codeExecuted = false; // Reset the flag if the mouse is not being pressed
    }
}

Oof! That's not going to work for p5play.

In ye old days of retro gaming, no wonder developers choose to keep the code simple instead. They programmed for 60fps and just had the games run slower in Europe lol. But it's +50 years later, so this problem really ought to be solved better in p5play.

Let's assume that we all like being able to check for equivalence with integers and that the time these numbers represent should be loosely equivalent, regardless of display rate. We also want a user's p5play program they developed using a 60fps display to run pretty much the same on any other display and vice versa.

Abstracting frames (revisited)

It seems abstracting frames is actually the way to go. But perhaps not in the way previously described:

like if everyone defined sketches as if it were 60fps, but sometimes the frame count is a fraction instead of a whole number

That'd be fine for frame rates higher than 60hz, but not lower. For example, Let's say p5play is running on a 50hz display, it'd need to convert 50fps frames to the 60fps equivalent by rounding.

for (let i = 0; i < 50; i++) {
    console.log(Math.round(i / 50 * 60));
}

0, 1, 2, 4, 5, 6, 7, 8, 10...

Obviously we're gonna be missing some integers, since 50 is less than 60. If when developing a game with my 60hz display, I check if the user has been pressing the mouse for 9 "frames", it would never be true if a player's draw loop rate is 50hz.

if (mouse.pressing() == 9) {
  // never going to run
}

So this abstract frame would need to be at least as large as the lowest frame rate p5play will support.

Let's say we want to support 30hz and 25hz, even Nintendo still uses such a low frame rate in their biggest games in order for the Switch to keep up. But then our abstract frame would be 1/25th of a second (40ms). Is that precise enough for detecting user input at high levels of play? Not really.

The current record for human button presses per second is around 10-15, achieved via the rolling technique used to play tournament level Tetris. That's one button press every 6ms, so 2.8 button presses per 60hz frame (16ms).

Latency and responsiveness are more important factors to consider. As a musician I can attest that low latency between an action, for example playing a key on a keyboard, and hearing a response is important. Humans can time actions with 10-20ms of precision when they are prepared to act. Higher than expected latency could make players think their game is running slow, even if it maintains a solid high frame rate visually.

Latency and Responsiveness

Is handling user input in the draw loop even a good idea? Most Unity developers do, putting input handling logic in the Update loop that runs once per frame. ChatGPT says for the majority of games, handling input in the Update method provides a good balance between simplicity, performance, and responsiveness.

In p5play, checking user input in the draw loop also enables users to do cool stuff like this:

if (kb.pressing('space') || mouse.pressing()) {
    sprite.color = 'green';
}

Visually there's no point in polling for user input more frequently than the delay between frames. But what about rhythm based games where ~20ms of audio delay is a concern? Fortunately there are already event based functions like mousePressed and keyPressed for that. Perhaps contros should let users define a input handling function that would run on every controller poll.

Better Name for "Abstract Frame"

On any device with a non-50hz display, the "abstract frame" or "standard frame" will be different from a real displayed frame. Arguably it could be better to give it an entirely different name then. As far as I'm aware there's no precedent for something like this.

ChatGPT suggests beat, pulse, tick, and quantum. Beat and pulse are too generic. "tick rate" is already commonly associated with server updates. "quant" sounds cool but "quantum" It doesn't really have anything to do with time and the implication that it's a really small unit makes it a misnomer. I'll have to think about this more.

I'm open to suggestions!

How about putting "base" or "ideal" at the beginning of the words?

ex)

baseFrequency
idealFPS

@ShiMeiWo I wouldn't want to use "ideal" cause it implies other fps rates are not ideal. I think baseFrequency is too long. It'd be nice to have a short name for the abstract frame.

Fixed vs Dynamic

With a Fixed-Time Step (Fixed Update Rate) approach, the game's logic updates occur at a consistent interval, typically every frame. This means that regardless of the frame rate, the game's logic progresses at the same pace. It offers consistency in gameplay speed but may result in less smooth experiences during performance fluctuations.

A Dynamic Frame Rate (Adaptive Time Scaling) approach adjusts the gameplay speed based on the frame rate. It ensures smoother gameplay experiences by synchronizing game mechanics with real-time rendering. By adapting to the frame rate, it prevents players from missing crucial frames and provides more consistent experiences across different hardware configurations.

Dropped frames

p5play v3 uses a fixed time step, this is a good approach for many games, since it is often more important to give players a chance to react to slowed down gameplay during performance bottlenecks than it is to essentially skip frames to keep pace with real time. Although for online games the opposite is true.

Should it be a goal in p5play v4 to separate the draw loop from the physics simulation, how should the abstract frame system handle dropped frames?

In the context of video games, dropped frames can occur when the hardware cannot keep up with the demands of rendering the game, resulting in skipped or missed frames in the animation sequence. This can lead to a less smooth and visually appealing gameplay experience, as animations can look choppy. Dropped frames are often an indication of performance bottlenecks that may need to be addressed through optimization techniques or hardware upgrades to ensure smoother gameplay.

Currently with p5play v3 if a game can't maintain 60fps, dropped frames are not counted in frameCount. Only after a frame is completed will p5play's post draw function run, which by default runs world.step and updates contact handling and input frame counters.

A big benefit of the fixed step system for developers is that it guarantees that the draw function will be run every time frameCount is incremented. So for example, there will always be a frame where mouse.pressing() == 8 if the user holds the mouse for 8 frames. I really like this aspect of the fixed time step system.

I will need to do some more research on this topic.

quinton-ashley / p5play