Dynamically restyle map based on framerate

elifitch commented 3 years ago

Motivation

Achieving a stable framerate can be a challenge, especially on lower end devices. One of the best ways you can do this is within the style. Reducing layer complexity, label collisions, aggressively filtering, so on and so forth.

This presents a nasty tradeoff: simplify a style and get a stable framerate on lower end devices, compromising the experience on better devices; or design the ideal style, which could make your map tremendously difficult to use for those folks on lower end devices.

Design

I propose including a map state lookup expression like ["zoom"] which returns the time to render the previous frame, or the maximum time to render a frame in this session. Something like ["frametime"]. A map designer could use ["frametime"] within a case expression to filter more and more features out of a layer in order to dynamically simplify a style and boost framerates on weaker devices, while preserving the style on higher end devices.

This design puts control in the hands of map designers, and allows them to shape a graceful, progressively "dehanced" style for lower end devices, while designing with an ideal style in mind.

If we determine that ["frametime'] leads to the style "flapping" (e.g. frame time is high => simplify style => frame time improves => complexify style => frame time is high => ) we could instead report the maximum frame time since the map began rendering, or the maximum frame time in the last ~240 frames or something like that.

Mock-Up

Example of how we could simplify a layer by adjusting its filter dynamically based on the frame time.

{
  "id": "my-road-network",
  "filter": [
    "case",
    // worse than 60fps remove tertiary roads
    [["frametime"], ">", 16.666]
    [
      "match",
      ["get", "class"],
      ["primary", "secondary"],
      true,
      false
    ],
    // worse than 30fps remove tertiary & secondary roads
    [["frametime"], ">", 33.333]
    [
      "match",
      ["get", "class"],
      ["primary"],
      true,
      false
    ],
    // 60fps or better show all roads
    [
      "match",
      ["get", "class"],
      ["primary", "secondary", "tertiary"],
      true,
      false
    ]
  ] 
}

Concepts

If we embrace this design, I think we'd need to figure out whether frame time or framerate makes sense. Frame time (e.g. 16.666ms) is a bit more flexible, framerate (e.g. 60fps) is more well-known outside of graphics programming circles.

We'd need documentation and recommendations around how to best use this expression as well.

karimnaaji commented 3 years ago

@elifitch , I made an RFC a while back on a similar topic around the idea of dynamically adjusting the rendering resolution (borrowing ideas from https://software.intel.com/content/www/us/en/develop/articles/dynamic-resolution-rendering-article.html and another product where I implemented a similar technique). I'll refer the internal RFC to this issue. Dynamic resolution could be more approachable when we start using the 2d render cache in both gl-js and native.

For styling I really like this proposal to have a level of granularity on a per layer basis, when things are dynamically changing based on framerate, the main technically challenging thing may be flickering, in prior dynamic resolution scaling approach I took I was keeping an history of frame time and only start scaling down or up if the framerate was decreasing for ~16 frames, this had a tendency to make things more stable visually and reduced abrupt changes. Similarly, layers could be fading on and off if the framerate has been detected to be bad enough for x frames.

Another approach could be style variants where the same style could have various configurations: e.g. low-end, high-end, ... The benefit of such an approach would be that not only we could flag certain layers to be on/off based on a particular variant, but we could also adjust particular techniques within the codebase to reduce their quality (Say we reduce terrain tesselation on low-end, but increase it on high-end, and also change various framebuffer resolution we use, etc...) but this would be a more static approach. I believe gmaps uses this technique selecting various technique quality code path based on the gpu vendor.

rreusser commented 3 years ago

Regarding flickering, a Schmitt trigger seems maybe applicable. In short, instead of a single value, the threshold is a lo/hi range with a single piece of state indicating whether you previously came from the high or low side. Thus, you don't change the quality until you surpass the threshold by some amount. Which is also the reason your thermostat doesn't flicker on/off when it's near the set point. (I suspect car headlights maybe also have a minimum switching time limit as well?)

I'm not sure if or how internal state like this could play with expressions though. (It could be accomplished in userland with access to the current quality state, but likely way too finicky and tricky for that to be a primary solution.)

mapbox / mapbox-gl-js