[feat] Please consider offering resolution control when in fullscreen

StoneCypher commented 2 years ago

Describe the problem

One important topic for game developers is resolution control.

Quite frequently, so as not to need multiple versions of art assets (cost and quality control,) or to present a pixel-identical version of a game to multiple players with different hardware over a network, or less frequently to manage render time on very complex screens, or even in some cases to simply achieve a certain aesthetic, it is preferable to control the resolution of a screen. Pixel-accurate stretches of images genuinely do not look the same as the screen in a different mode; there are too many opportunities for aliasing artifacts that one simply shouldn't have to spend time fighting; the impact on battery life for portable devices is significant.

I have a game that I would like to port using Tauri. Currently, I use DirectX and a native hosting of MSHTML to render, but that's tied to Windows, and since I've learned that Tauri targets Macs, Linux, and will soon target phones, I thought Tauri might be a nice platform to tackle portability

It's worth noting that this was the #1 request in DirectX 1 and 2 (DirectX hilariously didn't offer this until 3, because the Microsoft people were certain it was regressive; within a year of implementing it, more than 90% of AAA games were using it.)

This was also an extremely common help request in the Flash days, how to accomplish this, because the platform supported it but the relevant documentation wasn't very well written. I suggest this to support that it's well wanted by game developers.

On Windows, the relevant API for what I would like access to is ChangeDisplaySettingsExA, and the call should probably be made with the flag CDS_TEST.

Describe the solution you'd like

I'm kinda flexible here, and I don't use Tauri yet so this might be a braindead approach. Also, I'm prone to use Windows phrasing for things because I don't know the Mac phrasing and Linux has different names for this stuff for each window manager

But at first blush, I would think adding a member to /tooling/api/src/window.ts:WindowOptions to describe a screen's configuration, as well as a number to say which screen, might be the right thing to do?

Something like:

type DisplayConfig = {
    bitsPerPixel?: number | { 'at least': number } | false | undefined,  
    width?: number | false | { 'at least': number } | undefined,
    height?: number | false | { 'at least': number } | undefined,
    frequency?: number | { 'at least': number } | { 'multiple': number } | { 'exactly': number } | false | undefined,
    interlace?: boolean | undefined,
    textMode?: boolean | undefined,
};

type DisplayConfigPreferences = {
    allowContain?: boolean | 'horizontal' | 'vertical' | undefined,
    pixelRatio?: number | number[] | true | 'antialiased' | 'neighbor' | undefined 
};

type DisplayConfigRequest = DisplayConfig & DisplayConfigPreferences;

interface WindowOptions {
    // ... 

    display_target?: number | undefined,
    display_request?: DisplayConfigRequest | undefined
};

Admittedly, this approach would not allow single windows that spanned multiple screens, which will be a bummer to kiosk, hardwall, and dashboard vendors. However, that's actually relatively difficult due to things like difference in font scale between devices, and probably isn't necessary; besides, they could just create multiple windows.

Meanings:

bitsPerPixel:
- a number specifies an exact value.
- { 'at least': number } is useful for when you need 15 bits but a display reports as 16, or 24 as 30.
- undefined means to use the current device setting
- false means to not care at all, meaning even regression is fine
width (and equivalents for height):
- number: use this specific width
- false: do not consider width as a constraint when choosing a minimal screen sizing
  - if false is set for both width and height, one ought to end up with the smallest available screen; choose by pixel count
- { 'at least': number }: after filtering below the cut, do not consider width as a constraint when choosing a minimal screen sizing.
- undefined: use width specified in the parent object, window.WindowOptions.width; if that is also undefined, use the current screen width
frequency
- number: the closest frequency within 1% will be used (to cope with NTSC-rate 24/23.97, float error, etc); otherwise fail.
- { 'exactly': number }: only exactly the specified frequency, plus or minus Number.EPSILON, will be used; otherwise fail.
- { 'multiple': number }: the closest frequency within 1%, will be used. If none, integer multiple extensions (12hz would be satisfied by 24 on some monitors, or 60 on others, eg) would be considered. If none exist, fail.
- false: do not consider frequency a constraint; choose the lowest frequency at-or-above the current from what's eventually available.
- undefined: as false
interlace
- true: require interlacing to be on. these APIs generally do not offer control of leading edge.
- false: require interlacing to be off
- undefined: as false (ie, do not generally consider interlaced modes, even if they're better fits otherwise)
textMode
- true: bring up a good old fashioned console CLI text mode prompt. No, not a simulation.
- false: regular screen
- undefined: as false
allowContain
- true: if the resolution can't be matched, choose the next smallest fit, preferring smallest delta between requested and actual pixel count; expect the developer to handle their own letterboxing/pillarboxing
- false: require an exact fit, within the governance of pixelRatio
- horizontal: require an exact fit, within the governance of pixelRatio; prefer horizontal best-fits, as one would want for a side-scroller
- vertical: require an exact fit, within the governance of pixelRatio; prefer vertical best-fits, as one would want for a top-scroller
- undefined: as true
pixelRatio
- true: after finding the smallest screen resolution which can accommodate a request, scale the request by the largest integer factor that will still fit. Useful for putting a small resolution on any size of screen.
- number: specifies the single pixel scale ratio other than native which will be considered for match.
- number[]: specifies the list of pixel scale ratios other than native which will be considered for match.
- antialiased
- antialiased_fractional
- neighbor
- neighbor_fractional
- undefined: as true

The relevant Windows API offers a fullscreen field, but we omit that because it exists on the parent object, and unlike with height / width, there is no sensible situation in which the two differ

If the mode isn't available (CDS_TEST failure on windows,) then we need to know how to cope with a missing mode. An example of this might be if a developer asked for 1600x900 on a screen that does 1920x1080. Most screens of that size won't scale 1600x900. To that end we offer two features: contain and pixelRatio.

This suggests code in the following shape, probably with less annoying names than I chose:

type DisplayConfig = {
    bitsPerPixel?: number | { 'at least': number } | false | undefined,  
    width?: number | false | { 'at least': number } | undefined,
    height?: number | false | { 'at least': number } | undefined,
    frequency?: number | { 'at least': number } | { 'multiple': number } | { 'exactly': number } | false | undefined,
    interlace?: boolean | undefined,
    textMode?: boolean | undefined,
};

type DisplayConfigPreferences = {
    contain?: boolean | 'horizontal' | 'vertical' | undefined,
    pixelRatio?: number | number[] | true | 'antialiased' | 'neighbor' | undefined 
};

type DisplayConfigRequest = DisplayConfig & DisplayConfigPreferences;

function enumerateAvailableSPScreenConfigs(): DisplayConfig[] { ... }
function assignScreenConfig(targetScreen: number, config: DisplayConfig, allowAA: boolean) { ... }
function bestFitScreenConfig(desiredConfig: DisplayConfigRequest): DisplayConfig { ... }
function releaseScreenConfig(targetScreen: number) { ... }

Note: many APIs will return available hardware-scaled modes with non-square pixels. This approach requires that only resolutions with square pixels be considered. As such, a 4:3 mode on a 16:9 screen would be acceptable if letterboxed, but not acceptable if distorted. All major screen resolution APIs accommodate this.

This is relatively easily viewed as a decision tree: (editor)

Aside: why are these foo?: thing | undefined? Isn't that redundant?

No: foo?: thing doesn't allow explicit undefined members, and foo: thing | undefined doesn't allow omitted members. Both would be useful here.

Alternatives considered

A smaller implementation might start without the best-fitting API, without containment, without automatic pixel ratio seeking, without support for AA, neighbor, or fractional, drop support for textMode and interlacing, without support for at least or horizontal and vertical, in a pinch even drop frequency (not desirable!,) and have the end user implement. And that would be fine.

A successful minimal implementation IMO would look like this, and could be progressively expanded into the above:

type DisplayConfig = {
    bitsPerPixel?: number | undefined,  
    width?: number | undefined,
    height?: number | undefined
};

function enumerateAvailableSPScreenConfigs(): DisplayConfig[] { ... }
function assignScreenConfig(targetScreen: number, config: DisplayConfig) { ... }
function releaseScreenConfig(targetScreen: number) { ... }

Additional context

if you would like game developers, this is genuinely a pretty big deal

StoneCypher commented 2 years ago

For what it's worth I'm also willing to write most of this

amrbashir commented 2 years ago

I don't know if all of these options could be provided in a cross-platform manner. Windows has been leading in game dev for a while now so it is definitely possible to do it but for Linux and macOS, idk, might not have all of these features.

Tauri's window.fullscreen api is pretty basic since it is meant for Desktop Applications which usually just need to expand to cover the whole screen and not require resolution changes, however the underlying windowing library (TAO) does have extended fullscreen configurations, including multiple resolutions with multiple dpi and borderless fullscreen too.

I think a project like yours better be using WRY directly since IMO these requirements is rare for a Desktop/Mobile Application Framework and I don't think we should put much effort into it.

StoneCypher commented 2 years ago

This is all very easily done on Windows, Mac, and Linux. It's generally implied on phones. I don't think it would be much effort, especially given that you suggest TAO already does the heavy lifting.

I confess I'm a novice to the Tauri community, but, I would argue this is almost certainly far more impactful than the system tray, which you folks do support.

This is a deal breaker for much of gaming. Please consider helping out

amrbashir commented 2 years ago

This is all very easily done on Windows, Mac, and Linux.

I said Windows is possible because I mainly develop on/for Windows but I don't know about Linux and macOS. And if know the needed APIs to call, we always welcome PRs.

I confess I'm a novice to the Tauri community, but, I would argue this is almost certainly far more impactful than the system tray, which you folks do support.

Think of chat/mail apps like Discord, Slack, Mailspring which all need system trays to indicate new messages. Also think of background apps that only need trays to tweak a couple of options. System tray has proven it is a must-have for a Desktop framework.

I am not trying to demean the feature you're requesting but I don't think it is suitable for the framework and I don't think we would put any effort into this anytime soon. However, as I mentioned earlier, PRs are always welcome and I think we can expose TAO's fullscreen option as window.fullscreen_extended() and ofc related information like getting all monitors and different video modes.

As for other features, you need to implement them in TAO first before you can expose them in Tauri.

StoneCypher commented 2 years ago

you can just count the things in the windows store or steam using the apis

it's almost a hundred to one

amrbashir commented 2 years ago

I am not gonna go into statistics but if you want to see how much system trays is requested, you can check the issue tracker and suddenly your percentage will be the opposite.

And for the third time, PRs are welcome. You obviously stated that this sort of API is easy (according to you), so please contribute it. I am NOT trying to say this feature is not important, I am just saying it is a rare use-case for Tauri users and ONLY ONE person requested this (You).

JonasKruckenberg commented 2 years ago

I just want to chime in and say this is not personal and it's certainly not a lack of will. The feature you are proposing seems well thought out and game developers are a target audience we're trying to support to the best of our abilities. But here's the crucial part: best of our abilities. Our team is way to small for the awesome amount of use and love Tauri has been getting and we're all volunteers working nights and weekends on this. We simply have to prioritize what features to work on, sometimes a tough call, and right now our focus is elsewhere.

As amr stated though, we are always happy to accept contributions if you really need this feature help us out and implement it!

tauri-apps / tauri