[macOS] state of full screen

[artificiel]

there are 3 ways to make an OF window full screen on Mac:

1. OF fullscreen will "manually" place a window to cover the whole screen and hide menu and dock. while it works, neither the windowing system (GLFW) or the OS are made aware of the fullscreeness of the app (maybe they can deduce it, but it's not explicit). The process takes a couple of frames (about 0.5s on M1 Pro)

2. GLFW fullscreen with glfwSetWindowMonitor (currently not implemented in OF) assigns a window to a full monitor (optionally requesting a resolution). While GLFW paints the app full screen, it does not make the OS aware. The process in instantaneous. GLFW callbacks inform OF of the window resolution.

3. Appkit fullscreen a window can be made full screen with the green button in the menubar titlebar. this can be triggered with [cocoaWindow toggleFullScreen:nil]. the window animates as it fills the screen and moves to it's own "space", at the resolution of the screen. GLFW callbacks inform OF of the window resolution.

Since macOS 11, there has been optimisations going when putting an app in OS fullscreen. My experience with OF fullscreen is that I sometimes experience occasionally dropped frames (not raw frame rate, just a 2-3 frames gap here and there every 30-60 seconds). it needs critical conditions to be noticed. However switching to macOS fullscreen (ctrl-cmd-f on top of OF fullscreen or on it's own) I do not notice these gaps. it's hard to track/document, but it happens consistently enough for me to have developed this reflex.

Furthermore since macOS 14, on Apple Silicon, a "game mode" can be activated. it is a semi-opaque process that requires 2 things: being "OS fullscreen" and adding a plist entry to register as a "game". you are then granted this notification, plus a little menubar widget that allows you to turn it on and off:

What game mode means is not (yet) clearly defined in the apple docs, but an actual analysis of what happens has been made here. Evidently it seems like good parameters for a GL rendering app.

---

PROPOSAL: rewrite the macOS fullscreen functions around the OS native calls. toggleFullScreen is a toggle, but NSWindowStyleMaskFullScreen can reliably detect the current fullscreenness thus allowing the explicit ofSetFullScreen(bool) pattern. Additionally glfwSetWindowMonitor can be called on an OS-fullscreened window, making GLFW also aware of the fullscreenness of the window, benefitting from any compositing optimisation GLFW does when running as single window on a full monitor). Also rewrite getWindowMode so it is querying the OS's notion of things (instead of manually tracking it with a variable).

In short: make "ofSetFullscreen" equivalent of interacting with the green window button, or the "Enter full screen" menu option present in the Window menu. For UI consistency and potential optimisations.

Note that this does not prevent one to make a GL non-decorated window at the size of the screen, if for some reason the macOS fullscreen is undesired. macOS fullscreen works reliably on a M2 laptop with an external monitor (it fullscreens the monitor where the window is). however there is no mechanism to span multiple monitors (à la i3wm global fullscreen). incidentally in my linux usage I don't use ofSetFullScreen at all and instead rely on i3wm's config to set the OF window size to global fullscreen (or orchestrate whichever monitor I want the app to run in), and let OF adapt (i.e. OF does not know it is "full screen").

[artificiel]

[addition]: the content behaviour is similar wether the Setting "Screens have their own spaces" is turned on or off: the OF window takes over the monitor it is in. when "Screens have their own spaces", the other screens remain active; when they don't, they fade to black.

[artificiel]

related:

• 5158 (case 2 above)

• 6953

[ofTheo]

@artificiel thanks for diving into this and the detailed analysis

One of the main reasons fullscreen is the way it is, is that it allows us to go fullscreen on a second monitor without blanking the first monitor out ( as macOS's green button implementation does ).

This is used all the time for installations to say, have a non fullscreen main app window as a control panel and have a second window fullscreen across 1 or more monitors.

I assume 3) blanks out other monitors even if programatic?
Does 2) also do this, or does it just go fullscreen on the monitor selected and leave the other monitors as is?

[artificiel]

(3) programmatic or GUI user-driven blanks other monitors when "Screens have their own space" is false -- when activated, whatever content stays there.

(2) fullscreens on the passed monitor (can be a different one than the one the window is in); leaves the content in others as is (irrespective of Spaces).

and (side aspect) as spanning method might be useful. considering this setup:

where 0,0 is the upper-left of the window with the menubar. I can span that manually with ofGLFWWindowSettings::settings.decorated = false;, and ofSetWindowPosition(-1920,0); ofSetWindowSize(5760, 2160) and "automatically hide menubar" and "dock" in the Settings (of course the upper-left chunk of the render falls "nowhere"). it is equivalent to setting ofGLFWWindowSettings::settings.multiMonitorFullScreen to true.

that's what I was trying to express with my linux/i3wm reference above: for install-specific configs, I don't rely on ofFullScreen to trigger things inside-out but on the WM to organize the window rect (irrespective of the definition of "fullscreen", global or local as needed). in i3wm, meta-f will fullscreen the current window, and shift-meta-f will create a rect encompassing all screens and adjust the tile to the whole panorama (and i3 also allows conditional geometry based on window name which is great). but in general these configs are site-specific and hand-tuned.

my criticism of the current ofFullscreen functionality includes "user experience" from a typical macOS app standpoint:

1) it does not behave like the OS-offered mechanism — which is still present

2) OS fullscreen (and now Game Mode) optimisations are not activated by OF nor GLFW fullscreen

to come back to my anecdotic personal perceived performance (yet in theory supported by https://eclecticlight.co/2023/10/18/how-game-mode-manages-cpu-and-gpu/) I need to manually trigger OS fullscreen (title bar or menu item or OS keyboard shortcut) for the benefits to kick in.

so macOS fullscreen combined with "Displays have individual spaces" would cover your initial use case above. but i have a question:

a non fullscreen main app window as a control panel and have a second window fullscreen across 1 or more monitors.

as of now, how do you control things to determine where each window goes, and have the second window to fullscreen across more than 1 monitor? (but not all as multiMonitorFullScreen does?)

[ofTheo]

@artificiel - so I just tried the "screens have their own spaces" and that does allow fullscreen on one screen via

    auto cocoaWindow = (ofGetWindowPtr()->getCocoaWindow());
    [((__bridge NSWindow *)cocoaWindow) toggleFullScreen:nil];

to not blank out other screens. But it also has the downside of not allowing OF content to span multiple screens ( even not in fullscreen ).

So I think 3) might be a bonus function, but shouldn't be the default behavior.

For 2)

Calling this from ofApp::setup ( with two monitors - the second of which is 1920, 1200 ), fails.

    auto glfwWin = ((ofAppGLFWWindow *)ofGetWindowPtr())->getGLFWWindow();

    int count;
    GLFWmonitor** monitors = glfwGetMonitors(&count);
    glfwSetWindowMonitor(glfwWin, monitors[1], 0, 0, 1920, 1200, GLFW_DONT_CARE);

If I put it in ofApp::update at frame == 2 then it works, but as soon as I click outside of it to Xcode or another app it hides itself in the Dock? 🤷‍♂️

    if( ofGetFrameNum() == 2 ){
        auto glfwWin = ((ofAppGLFWWindow *)ofGetWindowPtr())->getGLFWWindow();

        int count;
        GLFWmonitor** monitors = glfwGetMonitors(&count);
        glfwSetWindowMonitor(glfwWin, monitors[1], 0, 0, 1920, 1200, GLFW_DONT_CARE);
    }

Going to try disabling screens have separate spaces and see if it is a bit more sane.

In general I am not opposed to making OF's fullscreen code more robust and with more options ( especially being able to select 2, 3, 4 monitors and say - go fullscreen across this rect ), but historically there is definitely a reason why it is the way it currently is. 🙂

Whatever changes we submit we will need a really good fullscreen testing app to go with it so we can test with different use cases and on different OSs.

[ofTheo]

Update: tried with 'screens have their own spaces' set to false and the glfw minimizing behavior doesn't change for 2) and 3) behaves as expected, blanking all other screens when app is fullscreen.

[artificiel]

for (2) I did this bound to a key and it seems to work with various resolutions (on the glfwGetPrimaryMonitor() but getting others is only a matter of figuring out in which one the window lies based on coordinates?):

 if (e.key == '[') {
    auto windowP = ((ofAppGLFWWindow *)ofGetWindowPtr())->getGLFWWindow();
    if (glfwGetWindowMonitor(windowP)) {
        glfwSetWindowMonitor(windowP, NULL, 200, 200, 320, 240, GLFW_DONT_CARE); // should restore orig
    } else {
        GLFWmonitor* monitor = glfwGetPrimaryMonitor(); // should find the one where the window lies
        const GLFWvidmode* mode = glfwGetVideoMode(monitor);
        glfwSetWindowMonitor(windowP, monitor, 0, 0, mode->width, mode->height, mode->refreshRate);
    }
}

the window does sneak into the dock on command-tabbing away (and reveals back with a click, but maybe there's a callback to respond to to undock upon getting focus? or maybe other options in GLFW to change the behaviour). the macOS behaviour of cmd-tabbing in/out is to slide the space to the side.

I'm trying to make a kind of matrix of concrete use cases (cannot do windows but at least macOS & linux(xorg+i3wm) and I'm still I'd like to get your specific input on your mention of:

a non fullscreen main app window as a control panel and have a second window fullscreen across 1 or more monitors.

how do you control things for the "second window" to adaptively span across more than 1 monitor?

I've done that with manual setWindowPosition/Shape, but it's totally depending on underlying desktop config. (in i3wm you have control on keybindings to do anything but you can also configure things so a window of a certain name (ex: myPlainApp) automatically gets fullscreened in a specific workspace (sort of equivalent to macOS space with "distinct spaces" enabled) or across ALL available spaces (width/height being the rectangle that fits all displays; note: that's a single GL context (proprietary Nvidia drivers)). the appeal of i3 is that you can be very precise about window geometry, and handle it at the WM level and rely on i3 config and let OF adapt. But there is no way to request a window to adaptively span 2 out of 3 displays (and that's why I'm asking the question on how you control that).

I would imagine making use of function such as ofGetAvailablePanoramaSize() to facilitate efforts like this, and perhaps provide a kind of model of how screens are organized, and knowing in which the running window is, etc. but these scenarios are sophisticated types of "fullscreeneness", and the initial impulse for this issue is I'm wondering about the naive ofToggleFullScreen() behaviour "for most users" and the UX coherence within the platform.

(also wondering why the multiMonitor support is tucked in the Settings instead of being more convivially accessible via a ofSetFullScreenMulti() kind of function? or ofSetFullScreen(bool fullscreen, bool multiMonitor = false)?)

in any case, the macOS game mode CPU/GPU priority thing kicks in only in Appkit fullscreen mode, and only 1 app can get the treatment — if you have multiple qualifying apps, only the mouse-owning one is activated the others are "paused" (it's not paused as in "not rendering" just that the mode itself is "paused" (confusing terminology if you ask me)):

so that means a single window in a single display, optionally with other normal (or fullscreen) windows on other displays in "distinct spaces" — at least for the time being; "serious gaming" will need to address multi monitor setups and perhaps the API will get less opaque.

NB: it is still to be measured how game mode concretely helps a machine "tweaked" for AV performance (where you probably organize things where you don't fight with random apps and 77 chrome tabs in the BG) but as I understand the M1/2/3 GPU multitasking it makes things more deterministic by assigning actual GPU cores, which would corroborate my experience with slow curves "dropping" frames here and there (even on an otherwise tranquil M2 pro machine). I'm still trying to find a way to take an objective/comparable measurement out of this.

[artificiel]

some semi-rigorous results. basic M1, macOS14.1; OF-git

60fps cpu-light rendering (~50% CPU 1 core) 1080p60, same compiled (released) bin. a couple of BG apps (Safari with a few tabs GitHub; idle Xcode, etc.

how activity monitors looks during tests:

NATIVE FULL SCREEN / GAME MODE: (one minute per line)
[notice ] mesures: 3600 min: 15.6501ms max: 17.7087ms mean: 16.6666ms var: 0.00622269ms
[notice ] mesures: 3600 min: 14.9016ms max: 18.4812ms mean: 16.6669ms var: 0.0293046ms
[notice ] mesures: 3600 min: 14.8365ms max: 18.4918ms mean: 16.6666ms var: 0.0238407ms
[notice ] mesures: 3600 min: 14.6621ms max: 18.7172ms mean: 16.6666ms var: 0.0270885ms
[notice ] mesures: 3600 min: 14.8487ms max: 18.5119ms mean: 16.6666ms var: 0.0284692ms
[notice ] mesures: 3600 min: 14.7867ms max: 18.6045ms mean: 16.6666ms var: 0.0348983ms
[notice ] mesures: 3600 min: 14.7366ms max: 18.6625ms mean: 16.6666ms var: 0.0287364ms
[notice ] mesures: 3600 min: 14.8183ms max: 18.5695ms mean: 16.6666ms var: 0.0253442ms
[notice ] mesures: 3600 min: 14.6687ms max: 18.7098ms mean: 16.6666ms var: 0.0374545ms
[notice ] mesures: 3600 min: 14.7983ms max: 18.5571ms mean: 16.6666ms var: 0.0441996ms

OF FULL SCREEN:  (one minute per line)
[notice ] mesures: 3600 min: 5.00238ms max: 29.2105ms mean: 16.6668ms var: 0.697817ms
[notice ] mesures: 3600 min: 5.45087ms max: 26.1817ms mean: 16.6666ms var: 0.416257ms
[notice ] mesures: 3600 min: 4.71112ms max: 31.2895ms mean: 16.6667ms var: 2.63432ms
[notice ] mesures: 3600 min: 8.70804ms max: 23.4159ms mean: 16.6668ms var: 0.174793ms
[notice ] mesures: 3600 min: 6.91925ms max: 27.3092ms mean: 16.6669ms var: 0.496939ms
[notice ] mesures: 3600 min: 4.99504ms max: 31.0159ms mean: 16.6674ms var: 2.64159ms
[notice ] mesures: 3600 min: 8.11483ms max: 24.2013ms mean: 16.6663ms var: 0.395399ms
[notice ] mesures: 3600 min: 7.66442ms max: 24.1905ms mean: 16.6672ms var: 0.244875ms
[notice ] mesures: 3600 min: 5.99079ms max: 24.4879ms mean: 16.6657ms var: 0.881461ms
[notice ] mesures: 3600 min: 7.19492ms max: 26.9372ms mean: 16.6667ms var: 2.51895ms

measurement code:

// .h
    std::vector<double> mesures_;
    std::optional<std::chrono::time_point<std::chrono::high_resolution_clock>> previous_time_;

// in update:
    if (auto t = previous_time_) {
        mesures_.push_back(((std::chrono::high_resolution_clock::now() - t.value()).count())/1000000.0f);
        if (mesures_.size()==60*60) mesure();
    }
    previous_time_ = std::chrono::high_resolution_clock::now();

// stats:
    auto mesure() {
        auto sz = mesures_.size();
        auto mean = std::accumulate(mesures_.begin(), mesures_.end(), 0.0) / sz;
        auto variance_func = [&mean, &sz](double accumulator, const double& val) {
            return accumulator + ((val - mean)*(val - mean) / (sz - 1));
        };
        auto zmin = *std::min_element(mesures_.begin(), mesures_.end());
        auto zmax = *std::max_element(mesures_.begin(), mesures_.end());
        auto variance = std::accumulate(mesures_.begin(), mesures_.end(), 0.0, variance_func);

        ofLogNotice("mesures") << sz << " min: " << zmin << "ms max: " << zmax << "ms mean: " << mean << "ms var: " << variance << "ms";
        mesures_.clear();
        previous_time_ = {};
    }

[artificiel]

PERCEPTUAL NOTE: the difference in jittering between the 2 modes is not very apparent — but it feels like a frame is dropped here and there with non-native fullscreen. also note I am looking at the render on a 75'' display; it makes a 1px line progression very sensitive.

But regardless of the perceived difference, the variance and extremes are much larger in non-native fullscreen / game mode.

[artificiel]

still digging and it seems to be related to this: https://github.com/glfw/glfw/issues/2249 (latest "interesting" comment: https://github.com/glfw/glfw/issues/2249#issuecomment-1719068093)

which is to say, "lower level" than OF, but something happens within the recent macOS GL handling that has negative side-effects which goes away in game mode.

also note that my animation problem seems exacerbated by the fact I'm using ofGetCurrentTimef() to synchronise the phase of paths, so if there is so jitter within frames, it jitters the animation -- not that there are not 60 frames in the second, but the time represented within each is not 1/60s (so ex: if an animation grows at the speed of 1px per ms, and for some reason i have 6ms interval between 2 updates, and a compensating 26ms before stabilizing at 16ms, I get a +6px, +26px, +16px progression, which is a perceivable inconsistency but hard to quantify (id' say "gritty"). I guess I need an ofGetNextFrameAnticipatedAppearanceTime() to time the "keyframes" to a real world period instead of the now apparently semi-random timepoint of update());

i don't still have a deterministic way of triggering things (apart from going in game mode which somehow "fixes" things, presumably triggering update() ASAP and not schedule it a bit later), but ofSetFrameRate() also interacts strangely (also: "discovered" ofSetTimeModeFixedRate but does not seem to help)

it might be a moment to think about what it would mean to dislodge OF from GL on macOS (imagining the backflips involved, esp on Apple GPUs). Also, Apple might anytime pull the plug on the long-deprecated APIs... maybe some hybrid approaches are possible (a native window backend, which still supports GL calls / shaders?)

[artificiel]

this is a stutter workaround implementable outside GLFW: https://github.com/google-deepmind/mujoco/commit/2f0fb1e4ef81ae4eb281e09fb67c6b36f27dc427

so until GLFW manages to implement the correct solution it might resolve the stuttering problem I experience, alleviating the game mode "requirement".

but (to stay on topic of this issue) it still leaves the question of the platform integration (green button, window and keystroke will go "native fullscreen" and not "of fullscreen").

my current idea is that if an ofSetFullScreen call would end up with a single monitor full screen (a deducible thing), it calls the native fullscreen function, and if not, it calls the current fullscreen code. kind of the best of both worlds, it stays coherent with users clicking the green button (who will get the expected WM/OS behaviour based on Spaces) but it allows fancier scenarios.

but this implies the stuttering is somehow fixed.

[danomatika]

Just to butt in: You can do "non-native" fullscreen in a "native" Cocoa app, so it's not like just because the application is on macOS means it has to obey the same behavior. I know this because I have actively fought post-Lion full screening due to the fact it takes over all of your monitors. This totally sucks when one screen is at a mixing desk and the second is 4K projector in front of a live audience.

I actually have a video player application written in swift which has an option to use "native" and "non-native" fullscreen. See ZirkVideoPlayer included with Zirkonium3. VLC also has a similar option in its settings.

ZirkVideoPlayer is a freeware app but not currently open source. In any case, here's the important bits which come down to setting some window flags manually then sizing the window to the current screen its on. Also keep a copy of the original windowed size to set it back to.

// MARK: Fullscreen

    /// full screen mode:
    /// * native: macOS native fullscreen using Spaces
    /// * custom: maximize to fill current screen
    enum FullScreenType {
        case native
        case custom
    }

    /// currently full screen?
    var fullScreen: Bool = false

    /// did the last full screen event use native mode?
    var fullScreenType: FullScreenType = FullScreenType.native

    /// frame when windowed, needed for custom fullscreen method
    var windowedFrame: CGRect = CGRect.zero

    /// connected via FirstResponder
    @IBAction override func toggleFullScreen(_ sender: Any?) {
        fullScreen = !fullScreen

        var type = (UserDefaults.standard.nativeFullScreen ? FullScreenType.native : FullScreenType.custom)
        if(!fullScreen && (type != fullScreenType)) {
            // make sure to return to windowed mode using the same full screen type
            type = fullScreenType
        }

        // toggle using full screen type
        switch(type) {
            case .native: // use Spaces
                super.toggleFullScreen(sender)
                break
            case .custom: // sizes to current screen
                // macOS 11+ does not allow setting .fullscreen directly
                if(fullScreen) {
                    windowedFrame = frame // store previous frame
                    level = .statusBar
                    let defaultScreen = UserDefaults.standard.defaultScreen
                    if(defaultScreen < 1 || defaultScreen > NSScreen.screens.count) {
                        // use current screen
                        setFrame(screen!.frame, display: true, animate: false)
                    }
                    else {
                        // use specific screen
                        setFrame(NSScreen.screens[defaultScreen-1].frame, display: true, animate: false)
                    }
                    if #available(macOS 11.0, *) {
                        NSApp.presentationOptions.insert([.autoHideMenuBar, .autoHideDock])
                        styleMask.remove([.titled])
                    }
                    else {
                        styleMask.insert(.fullScreen)
                    }
                }
                else {
                    if #available(macOS 11.0, *) {
                        styleMask.insert([.titled])
                        NSApp.presentationOptions.remove([.autoHideMenuBar, .autoHideDock])
                    }
                    else {
                        styleMask.remove(.fullScreen)
                    }
                    level = .normal
                    setFrame(windowedFrame, display: true, animate: false)
                }
                break
        }
        fullScreenType = type
        NotificationCenter.default.post(name: Notification.Name("PlayerFullScreen"), object: nil)
        printDebug("Player: fullScreen \(fullScreen)")
    }

I would say the points about performance issues with GLFw's window setup are important but I wouldn't labor too much on making sure all OF apps behave like desktop productivity applications.

[danomatika]

Sorry, don't get me wrong. I think supporting the native fullscreen behavior is a good idea, especially for a single window fullscreen application which could utilize the new game mode. I just would not want us to throw away the "old fullscreen" in the process and I think the current behavior should stay the default for OF applications. IMO choosing the behavior could be a platform-specific hint such as a flag and not really a function.

[artificiel]

yes, definitely there are situations where a "partial" fullscreen is desirable but — notwithstanding this performance issue — it is hard to express within OF's current boolean fullscreen approach (hence my question to @ofTheo as to how they control their case of 1 control window + >1 spanned "fullscreen" outputs).

in any case the GLFW issue linked above confirms the stuttering, and also confirms it's not an OF issue per se. where I'm diagonally coming from is that going native fullscreen in game mode solves the problem.

and note that for a while (circa 2017 not sure of macOS version) I've had the habit not to use ofFullScreen but manually placed an undecorated window with dock/menu hidden as it generally provided better performance. but now even such a manual setup stutters. to be clear, the stuttering is not only occasional dropped frames (even under low CPU load), but is due to the fact that update() timing jitters, and being not isochronous, ofGetElapsedTimef() inside update() cannot be relied upon for key framing animations.

i am not sure what you mean by:

making sure all OF apps behave like desktop productivity applications.

maybe I'm not sure what "desktop productivity" means, but here my concern is that a rendering in OF fullscreen stutters, and hitting ctrl-cmd-f on the same window (OS-fullscreening+gamemode an OF-fullscreened window) removes the stuttering. I expect at least either (a) OF-fullscreen not stuttering in the first place, or (b) have a means to activate OS-fullscreen from within OF.

to summarize:

1. OF should not stutter/jitter a. wait on GLFW? b. implement an external fix like the google one above? or perhaps your swift model above if it addresses the issue c. bypass GLFW and implement a lower level backend? d. use native fullscreen / game mode (acceptable for single monitor or "distinct spaces") e. circumvent update() jitter with something like ofGetNextFrameAnticipatedAppearanceTime()

2. OF should be more malleable vs what "fullscreen" means a. provide a mechanism to trigger the native fullscreen (maybe GLFW will come around to it?) b. if a user uses the OS functions (green button, ctrl-cmd-f) net result should be identical to (2a) c. provide a way to trigger "total panorama fullscreen" d. provide some introspective tools to organise partial fullscreen

in terms of use cases there are not dozens of possibilities:

• if you have a single monitor, fullscreen == native fullscreen and gain game mode
• if you have distinct Spaces enabled; same as above
• if you want to span more than one monitor, fullscreen becomes ambiguous; more control is needed, and will probably rely on a flavour or the current OF fullscreen mechanism, which requires a stutter fix different from game mode.

[ofTheo]

@artificiel - so I believe our current multi fullscreen approach is similar to what you do, disable decoration and manually span the displays we want.

I think this could be a cross platform feature that we could add to ofAppGLFWWindow, I think something like:

ofAppGLFWWindow::setFullscreenWithRect( ofRectangle & fullscreenRect );

And maybe a better way to query the rects of all monitors to easily build a spanning rect to pass to the function.

I think we could also have an option in ofGLFWWindowSettings like bool useNativeFullscreen or even bool useMacOSNativeFullscreen to do the native mode by default with setFullscreen.

if you have a single monitor, fullscreen == native fullscreen and gain game mode

this could be an option for sure. though it just applies to macOS - not sure if there would be any downsides.

if you have distinct Spaces enabled; same as above

sure - again, would be good to think about possible edge cases

For the stuttering. Maybe we just host a fork until it's fixed? We've done this in the past with apothecary.

[artificiel]

For the stuttering. Maybe we just host a fork until it's fixed? We've done this in the past with apothecary.

there is a tentative fork https://github.com/RustyMoyher/glfw_MacVsyncFix but as I read the issues it's not a solved problem. the google fix works in the app space, but seems involved as it basically rewrites vsync https://github.com/google-deepmind/mujoco/commit/2f0fb1e4ef81ae4eb281e09fb67c6b36f27dc427 but also maybe it does more than we need so can be somehow lifted?

jitter or not, as far as animation goes ofGetNextFrameAnticipatedAppearanceTime()(name TBD) could not hurt -- there might be other conditions where the execution of update() is not isochronous (turning ofGetElapsedTimef call intervals irregular), and with phase-based animation the objective is generally to paint the image for the moment it will appear (skating to where the puck will be, so to say). [like offline rendering one could rely on ofGetFrameNum() * frame_dur but that implies never a drop frame — or fall out of sync].

based on the timing stats I posted above, it would solve the problem in terms of net result (I don't mind if update()/draw() is irregular as long as I drew the image for the future moment it will appear and not for the moment ofGetElapsedTimef() in update() happened to occur). (and as long as the frame gets drawn within the 1/60s, which seems to be mostly reliable).

(I'm still tweaking it a bit but will share a test app to validate this jitter/stutter thing — it's surprisingly annoying to test, but I'm getting consistant readings)

I believe our current multi fullscreen approach is similar to what you do, disable decoration and manually span the displays we want.

yes, but the OF multifullscreen is more adaptive (my approach relies on pre-known specific display arrangement — which is OK for installation- and performance-specific apps; not OK for a distributed app or ones that need to adapt to different conditions. I was mentioning my approach in response to @danomatika the make sure it was clear I was not arguing for a pure macOS UX).

ofAppGLFWWindow::setFullscreenWithRect( ofRectangle & fullscreenRect );

sounds good, and it should cover the needs of any "advanced" usage, as long as one figures out the rectangle spec. a getMaximumFullScreenRect (similar to current multi calculations) would be a start. and then when calling fullscreen, check if the rect matches what "native full screen" would do (I guess that's deducible), and if so call native on fullscreen() [edit to add: and if you have multiple monitors and don't want them fade to black, turn on "distinct spaces"], otherwise call manual code [with "distinct spaces" off if you want to span monitors].

it then ignores glfwSetWindowMonitor — in any case the fullscreen handling code is already platform-ifdef'd, but it would be good to verify what windows does with that call? (linux/i3wm does not seem to care about it — at least on X11/nvidia-proprietary; don't know about other linux WMs / drivers / nor wayland).

but I'm still curious to know how you control your scenario of 1 control window floating in a monitor, plus a spanned fullscreen render in 2+ extra monitors. doing that "adaptively" (without explicit setPosition etc) requires knowing about individual display sizes and relative positions?

[danomatika]

The fullscreen handling code in ZirkVideoPlayer may not be the solution for OF but could be useful compare with. For instance, the behavior with the fullscreen window hint changes with macOS 11, so I actually no longer use it.

The Google fix is doing kind of what I first thought of: using an NSDisplayLink which is basically a special high-performance timer that fires when the screen is updated. It has timing that is much more reliable than a regular NSTimer and is easier to use than a low-level Mach timer, so I find it useful for triggering background processing like processing incoming OSC messages.

In this case, we could apply a patch to GLFW in the Apothecary formula, as an option, or as Theo suggests use a fork.

@artificiel:

i am not sure what you mean by:

making sure all OF apps behave like desktop productivity applications.

maybe I'm not sure what "desktop productivity" means

Sorry, that was perhaps not fair. I was implying that you may be too focused on making the fullscreen button work the same as applied to apps which are different than a typical OF app, ie. a web browser or text editor. IMO it's not worth trying to enforce this at a loss of the original windowless fullscreen approach between native and non-native fullscreen modes. Also, what I remember the last time I looked, the green fullscreen button is hardwired to always uses the native fullscreen. The event cannot be overridden or intercepted like with the menu item, so I suggest not bothering and it's a compromise we may have to live with. (I have been around this kind of block before and fixing that last 1% is a marginal gain leading to open source burnout.)

I was mentioning my approach in response to @danomatika the make sure it was clear I was not arguing for a pure macOS UX). 👍

Similarly, I also wouldn't worry too much about fleshing out a multi-window environment. There was an addon which attempted to do that some years ago and I believe worked well at first but maintenance on custom code due to system changes becomes too much. I would be hesitant to expand things too much in this area unless it's something handled via a cross-platform library upstream (GLFW, SDL, etc). There is just too much work keeping it up to date that it's best shared.

it might be a moment to think about what it would mean to dislodge OF from GL on macOS (imagining the backflips involved, esp on Apple GPUs). Also, Apple might anytime pull the plug on the long-deprecated APIs... maybe some hybrid approaches are possible (a native window backend, which still supports GL calls / shaders?)

The answer to this is provide a Metal backend, if anyone cares to spend the time...

EDIT: Interestingly, from what I have read before, the current OpenGL implementation Apple provides actually wraps Metal. It would be nice if that was open source.

[ofTheo]

but I'm still curious to know how you control your scenario of 1 control window floating in a monitor, plus a spanned fullscreen render in 2+ extra monitors. doing that "adaptively" (without explicit setPosition etc) requires knowing about individual display sizes and relative positions?

Ahh well in our case we are usually dealing with fixed setups that don't change. So we just set a fixed rectangle based on a known offset and known screen sizes 🙂

In theory we could do something like

setFullscreen("1,2,3");

And let the ofAppGLFWWindow figure out the offsets and total size, but I think that might get too involved and its probably better to let users query the monitor info and then make a single rect from that info.

[ofTheo]

The answer to this is provide a Metal backend, if anyone cares to spend the time... EDIT: Interestingly, from what I have read before, the current OpenGL implementation Apple provides actually wraps Metal. It would be nice if that was open source.

We would be able to do this via WebGPU - the current implementations are pretty good and they run everything on Metal on macOS. It would be a big lift to integrate with OF ( mostly with the GL specific classes: ofMaterial / ofTexture / ofFbo etc ) but I've already got it working once before ( in a simple way ) and I know there is interest there. Probably good to start an issue / discussion on it. The webgpu-native project and the c++ wrapper should be all we need: https://github.com/eliemichel/WebGPU-Cpp

[artificiel]

yes: time to start better issues. this one became a progressive research project that mixed separate actual problems/solutions. so let's presume I'm writing the issue with what was learned:

recent changes to macOS introduces jitter and stutter in the timing of OF rendering. it is at least in part inherited from GLFW; probably other aspects comme OS-wide behaviours in regards to CPU and GPU priority. this is exacerbated by using vsync (an official GLFW bug; circumventable but requires involved implementation (cf google deep-mind), and reduced by going into OS-fullscreen+gamemode (with the macOS-imposed limitation of being limited to a single screen).

the jitter is sometimes large enough to drop frames, even under very light CPU loads. and when not dropping frames, calling ofGetElapsedTimef() in update() or draw() does not happen in a phase-locked iscohronous relation with the appearance of images on screen, making ofGetElapsedTimef() unusable for phase-based animations.

my original motivation was based on perceptually noticing a "better" rendering result with fullscreen game mode. with the following tests I've gone from a subjective assessment to an objective one. the process led to gain awareness the GLFW/vsync bug(s), as well as determining that GL apps are somehow "sandboxed" by recent macOS quartz updates, and the only way to currently get consistant optimal results is by running fullscreen-game mode, no vsync.

trying things with the GLFW fullscreen method (currently not implemented by OF) does not yield perceivable differences vs OF fullscreen (makes sense, considering GLFW has not solved the vsync bug yet).

To complete the picture here are 6 screenshots of a test app that draws the timing interval between frames. there are 6 cases:

• OF vs OS fullscreen
• vsync vs no sync
• game mode vs no game mode (in OS fullscreen)

[note that there is another GLFW bug that initially reports the vsync at 120Hz which breaks things if you run vsync without specifying a framerate — I've decided not to enter that additional rabbit hole, the tests are made with ofSetFrameRate(60) to normalize things on that aspect]

ANALYSIS: we notice that the worst case is OF fullscreen + vsync (the actual "default"). disabling vsync helps and we see similar behaviour with OF fullscreen and OS fullscreen no game-mode. then with game mode, we see that vsync hinders timing a bit, and that with no vsync + game mode we attain almost perfect timing.

[ofTheo]

Wow - @artificiel - this is quite an investigation. Definitely feel free to open more focused issues, but I meant moving the webgpu to a discussion ( in case it sounded like moving/closing this issue ).

One other factor to consider is if we change any of the default fullscreen modes for macOS, how might these work on non ARM or pre macOS 13 versions.

So it might be good to do the things we discussed above for single monitor / separate spaces as a behavior you can change via the settings incase there is a negative effect on older macOS versions.

[artificiel]

yes a bit more involved that I initially thought but it is good to have objective measurements (and confirm I'm not imagining things — "it feels better" is not a great argument).

I will split the content into a few issues that are addressable independently and will take note of your non-ARM / pre13 (although the [cocoa fullscreen] method is implemented since macOS 10.7 Lion, which is quite a way back).

in essence the core problem is the GLFW sync bug (macOS13+), which unfortunately is the default OF behaviour. game mode (macOS14) is the quick way to level out most negative side effects. I guess that makes macOS13 an undesirable OS version.

[artificiel]

this issue distilled out into 6 mostly freestanding issues, working as a whole to solve the initial performance problem:

• 7758

• 7759

• 7762

• 7761

• 7763

• 7760

and is required for the RenderTimingTests app to be shared:

• 7757

[dimitre]

Great insights. We can maybe create a discussion so each topic can be individually commented.

[artificiel]

how about discussing the topics in the associated issues listed above? they have been specifically cut out as they can be independently focused upon (this issue has been closed because what initially seemed like a simple switch revealed deeper issues and modular elements of "a solution").

[dimitre]

Yes, it is only a suggestion about the format. issues are linear like a big chat. and in discussions you can comment each one individually so it is easier to follow up now or later. As this discussion has the potential to grow like a tree

[artificiel]

ah yes, exactly: "like a tree" is what happened here as things got analyzed, and why I'm splitting up. for instance the GLFW bug has nothing do do, intrinsically, with macOS game mode (even if the latter more or less "solves" the former), and with "native full screen" (even if it's required for game mode).

but I find the GitHub discussions are more like echo chambers where things sort of fade out without clear resolution.

there is already a lot of material up here, I will be happy to distill/re-contextualize as answers to questions/comments in the distinct issues threads.

[ofTheo]

For me personally discussions are good for more big picture stuff and the issues are better for narrower topics which can be acted on.

For example: WebGPU as a discussion topic makes sense.

This whole issue also could have been a discussion but in the end breaking it out into issues which we can PR against makes sense. So fine to leave as is.