emacs 28: flicker present (extra redraws?) on darwin, not on homebrew

[mjlbach]

Describe the bug This is a weird one, but I wanted to document it in case others are having the same problem while I try to find a fix.

I'm sorry the reproduction steps are so specific, I'll keep the issue updated as I remove variables. I'm wondering if this is due to the macos framework we are linking against.

To Reproduce Steps to reproduce the behavior:

1. Install native-comp (33b8ce865fcfd58538ae2d7c3fff04998fcd3330) or emacsGit from the emacs-overlay and via d12frosted's emacs-plus tap for homebrew. The issue is not reproducible on emacs 27 or emacsMacport 27 (for me).
2. Install doom-emacs
3. Open an org file
4. Trigger save, or insert-line (o)
5. Trigger redraw-display introduces the flicker on both nix and brew (as expected)

This leads me to believe the nix version is triggering redisplay. Which I can't explain, as none of the hombrew patches seem to touch pertinent parts of the display code.

Expected behavior The behavior to match across homebrew and nix packaging.

Screenshots Via homebrew:

https://user-images.githubusercontent.com/13316262/105077063-cede1600-5a40-11eb-924d-fb4bf6e32ec7.mov

Via nix:

https://user-images.githubusercontent.com/13316262/105077169-f46b1f80-5a40-11eb-8b33-917237044360.mov

Additional context Add any other context about the problem here.

Notify maintainers Very niche intersection, I'm guessing:

@jwiegley @matthewbauer

Metadata

```

My flake.lock

{ "nodes": { "LS_COLORS": { "flake": false, "locked": { "lastModified": 1609339936, "narHash": "sha256-6DnZgWXlQ1+focJGvhlVvgo97owDCj5w2zydF2ZiV8Q=", "owner": "trapd00r", "repo": "LS_COLORS", "rev": "e91cc9cc69f6c4780f03b121bc633569742de7cd", "type": "github" }, "original": { "owner": "trapd00r", "repo": "LS_COLORS", "type": "github" } }, "emacs-git": { "flake": false, "locked": { "lastModified": 1609982357, "narHash": "sha256-t6FLgsdi7PGzipX+Sc3pQa0x1o5kTYnEns2wP4X0eTM=", "owner": "emacs-mirror", "repo": "emacs", "rev": "9b31802e2d13cf8478c23d651741b54ffd4b984b", "type": "github" }, "original": { "owner": "emacs-mirror", "repo": "emacs", "type": "github" } }, "emacs-nativecomp": { "flake": false, "locked": { "lastModified": 1609944778, "narHash": "sha256-upLKdC2XMRiCc5cDt/t6hl9ZqK/3KPTJ605aKZIHrTM=", "owner": "emacs-mirror", "repo": "emacs", "rev": "33b8ce865fcfd58538ae2d7c3fff04998fcd3330", "type": "github" }, "original": { "owner": "emacs-mirror", "ref": "feature/native-comp", "repo": "emacs", "type": "github" } }, "emacs-overlay": { "inputs": { "emacs-git": "emacs-git", "emacs-nativecomp": "emacs-nativecomp", "emacs-pgtk": "emacs-pgtk", "emacs-pgtk-nativecomp": "emacs-pgtk-nativecomp", "emacs-unstable": "emacs-unstable", "exwm": "exwm", "flake-compat": "flake-compat", "flake-utils": "flake-utils", "gitignore.nix": "gitignore.nix", "nixpkgs": "nixpkgs", "pre-commit-hooks.nix": "pre-commit-hooks.nix", "xelb": "xelb" }, "locked": { "lastModified": 1610008238, "narHash": "sha256-N9Ynpg0n59m/DEIOoiwjo9mohtrWCARc+pTtiAL/sYY=", "owner": "mjlbach", "repo": "emacs-overlay", "rev": "d62b49ac651e314080e333a7e1f190877675ee99", "type": "github" }, "original": { "owner": "mjlbach", "ref": "feature/flakes", "repo": "emacs-overlay", "rev": "d62b49ac651e314080e333a7e1f190877675ee99", "type": "github" } }, "emacs-pgtk": { "flake": false, "locked": { "lastModified": 1606038698, "narHash": "sha256-zNk5JrwQMsS/tve+F8iPybRl3GYJPO0/hVuldRI7x7A=", "owner": "masm11", "repo": "emacs", "rev": "3df4ca451d41a5f1036713277ef55ca9734c6fa7", "type": "github" }, "original": { "owner": "masm11", "ref": "pgtk", "repo": "emacs", "type": "github" } }, "emacs-pgtk-nativecomp": { "flake": false, "locked": { "lastModified": 1609974636, "narHash": "sha256-QB8pZBGBXhj74/OwOW9BXXQjIJL+CjcRxCWgT4OPZqM=", "owner": "flatwhatson", "repo": "emacs", "rev": "bd02c8c8fd51480e4ee05cbc299ca68ae0dc2ec2", "type": "github" }, "original": { "owner": "flatwhatson", "ref": "pgtk-nativecomp", "repo": "emacs", "type": "github" } }, "emacs-unstable": { "flake": false, "locked": { "lastModified": 1609857945, "narHash": "sha256-ASXqq6/677ftCTiIXJqpUx6irgNWaFLOnvlLmD6ih/Y=", "owner": "emacs-mirror", "repo": "emacs", "rev": "149d64bbb2b46f63c759fe4754bdf90eb6f2a3cc", "type": "github" }, "original": { "owner": "emacs-mirror", "ref": "emacs-27", "repo": "emacs", "type": "github" } }, "exwm": { "flake": false, "locked": { "lastModified": 1594512000, "narHash": "sha256-PHMHtTbKpKtFGGGtMh2SbKoK+8avh3CiDeW5jXP+TDM=", "owner": "ch11ng", "repo": "exwm", "rev": "0368127976bda29d35eed788edfe74644ecd3845", "type": "github" }, "original": { "owner": "ch11ng", "repo": "exwm", "type": "github" } }, "flake-compat": { "flake": false, "locked": { "lastModified": 1606424373, "narHash": "sha256-oq8d4//CJOrVj+EcOaSXvMebvuTkmBJuT5tzlfewUnQ=", "owner": "edolstra", "repo": "flake-compat", "rev": "99f1c2157fba4bfe6211a321fd0ee43199025dbf", "type": "github" }, "original": { "owner": "edolstra", "repo": "flake-compat", "type": "github" } }, "flake-compat_2": { "flake": false, "locked": { "lastModified": 1606424373, "narHash": "sha256-oq8d4//CJOrVj+EcOaSXvMebvuTkmBJuT5tzlfewUnQ=", "owner": "edolstra", "repo": "flake-compat", "rev": "99f1c2157fba4bfe6211a321fd0ee43199025dbf", "type": "github" }, "original": { "owner": "edolstra", "repo": "flake-compat", "type": "github" } }, "flake-utils": { "locked": { "lastModified": 1609935452, "narHash": "sha256-McwA/tAnnS8LaAdEoONBsdKFHuOpHMxKqpMCU1wL7H4=", "owner": "numtide", "repo": "flake-utils", "rev": "8088c6dbe86a7e6e6396c83e43020e8a1edb08d5", "type": "github" }, "original": { "owner": "numtide", "repo": "flake-utils", "type": "github" } }, "flake-utils_2": { "locked": { "lastModified": 1610051610, "narHash": "sha256-U9rPz/usA1/Aohhk7Cmc2gBrEEKRzcW4nwPWMPwja4Y=", "owner": "numtide", "repo": "flake-utils", "rev": "3982c9903e93927c2164caa727cd3f6a0e6d14cc", "type": "github" }, "original": { "owner": "numtide", "repo": "flake-utils", "type": "github" } }, "gitignore": { "flake": false, "locked": { "lastModified": 1594969032, "narHash": "sha256-nbZfz02QoVe1yYK7EtCV7wMi4VdHzZEoPg20ZSDo9to=", "owner": "hercules-ci", "repo": "gitignore.nix", "rev": "c4662e662462e7bf3c2a968483478a665d00e717", "type": "github" }, "original": { "owner": "hercules-ci", "repo": "gitignore.nix", "type": "github" } }, "gitignore.nix": { "flake": false, "locked": { "lastModified": 1594969032, "narHash": "sha256-nbZfz02QoVe1yYK7EtCV7wMi4VdHzZEoPg20ZSDo9to=", "owner": "hercules-ci", "repo": "gitignore.nix", "rev": "c4662e662462e7bf3c2a968483478a665d00e717", "type": "github" }, "original": { "owner": "hercules-ci", "repo": "gitignore.nix", "type": "github" } }, "home-manager": { "inputs": { "nixpkgs": [ "nixpkgs" ] }, "locked": { "lastModified": 1610791052, "narHash": "sha256-2sqrLo1O0OmutNyPZTg5lXDNPDgjcrlvAkQbo7pFUUY=", "owner": "nix-community", "repo": "home-manager", "rev": "8127799f79ee96129b295d78294f40a54078131f", "type": "github" }, "original": { "owner": "nix-community", "repo": "home-manager", "type": "github" } }, "neovim-nightly": { "flake": false, "locked": { "lastModified": 1610913682, "narHash": "sha256-06nfbvwe1lCvOnA/RnLSonWASfbowMFKVlscdzWzopc=", "owner": "neovim", "repo": "neovim", "rev": "702208daa6553ecfc3939c3c6e9fcd94fdd5aeba", "type": "github" }, "original": { "owner": "neovim", "repo": "neovim", "type": "github" } }, "neovim-nightly-overlay": { "inputs": { "flake-compat": "flake-compat_2", "flake-utils": "flake-utils_2", "gitignore": "gitignore", "neovim-nightly": "neovim-nightly", "nixpkgs": "nixpkgs_2", "pre-commit-hooks": "pre-commit-hooks" }, "locked": { "lastModified": 1610958998, "narHash": "sha256-tFJbQ3I1zu+XtXenVcGkP6PdRw56I1vpISCeQi1ykUY=", "owner": "nix-community", "repo": "neovim-nightly-overlay", "rev": "6ec8a36a5ff439fe312eef8ae8c4c29f4afa6127", "type": "github" }, "original": { "owner": "nix-community", "repo": "neovim-nightly-overlay", "type": "github" } }, "nixpkgs": { "locked": { "lastModified": 1609866355, "narHash": "sha256-TMzb8nMePC4OusDde+xv8AIgbNYuFku/dlLkPO/hifM=", "owner": "nixos", "repo": "nixpkgs", "rev": "d9dba88d08a9cdf483c3d45f0d7220cf97a4ce64", "type": "github" }, "original": { "owner": "nixos", "ref": "nixos-unstable", "repo": "nixpkgs", "type": "github" } }, "nixpkgs_2": { "locked": { "lastModified": 1610942247, "narHash": "sha256-PKo1ATAlC6BmfYSRmX0TVmNoFbrec+A5OKcabGEu2yU=", "owner": "nixos", "repo": "nixpkgs", "rev": "7d71001b796340b219d1bfa8552c81995017544a", "type": "github" }, "original": { "owner": "nixos", "ref": "nixpkgs-unstable", "repo": "nixpkgs", "type": "github" } }, "nixpkgs_3": { "locked": { "lastModified": 1610942247, "narHash": "sha256-PKo1ATAlC6BmfYSRmX0TVmNoFbrec+A5OKcabGEu2yU=", "owner": "nixos", "repo": "nixpkgs", "rev": "7d71001b796340b219d1bfa8552c81995017544a", "type": "github" }, "original": { "owner": "nixos", "ref": "nixpkgs-unstable", "repo": "nixpkgs", "type": "github" } }, "pre-commit-hooks": { "flake": false, "locked": { "lastModified": 1609781048, "narHash": "sha256-RTu4NDdAgwv83aFFJDnUkQFl/7giDSeEtQ/N17xxiXY=", "owner": "cachix", "repo": "pre-commit-hooks.nix", "rev": "c7e3896e35ceea480a7484ec1709be7bdda8849d", "type": "github" }, "original": { "owner": "cachix", "repo": "pre-commit-hooks.nix", "type": "github" } }, "pre-commit-hooks.nix": { "flake": false, "locked": { "lastModified": 1609781048, "narHash": "sha256-RTu4NDdAgwv83aFFJDnUkQFl/7giDSeEtQ/N17xxiXY=", "owner": "cachix", "repo": "pre-commit-hooks.nix", "rev": "c7e3896e35ceea480a7484ec1709be7bdda8849d", "type": "github" }, "original": { "owner": "cachix", "repo": "pre-commit-hooks.nix", "type": "github" } }, "root": { "inputs": { "LS_COLORS": "LS_COLORS", "emacs-overlay": "emacs-overlay", "home-manager": "home-manager", "neovim-nightly-overlay": "neovim-nightly-overlay", "nixpkgs": "nixpkgs_3" } }, "xelb": { "flake": false, "locked": { "lastModified": 1595116800, "narHash": "sha256-7Xr5UJyVxU8RatC8fo1x7WUpuY3qf/1izRMEMzR8dIM=", "owner": "ch11ng", "repo": "xelb", "rev": "df102a5773b37cec154e795a17a8513144dde643", "type": "github" }, "original": { "owner": "ch11ng", "repo": "xelb", "type": "github" } } }, "root": "root", "version": 7 } ```

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: emacs

[ethancedwards8]

I also have this issue. I'm not running nativecomp, just pkgs.emacs

[inscapist]

Just want to inform this thread that I have tested a fix by modifying the CFLAG here

It has been discussed on reddit for posterity.

[matthewbauer]

Just want to inform this thread that I have tested a fix by modifying the CFLAG here

It has been discussed on reddit for posterity.

We removed that flag in https://github.com/NixOS/nixpkgs/commit/5efbcf8d7eef8a6380dafb9fbfb6420b1e051da9. Is that enough to fix things, or is 101200 necessary?

[inscapist]

@matthewbauer let me get back on this later today.

[inscapist]

I am afraid the flicker issue persists. Tested on both 27.2 and 28 (with overlay)

[matthewbauer]

Ok. I think a pr setting MAC_OS_X_VERSION_MAX_ALLOWED=110200 would be accepted! I guess the only question would be whether it breaks pre big sur systems.

[inscapist]

I can try it on my wife's Macbook Air. I will revert this weekend!

[colonelpanic8]

I'm getting pretty flickery behavior in nixos. Is it possible that this may not be a macos specific issue?

[Atemu]

No, this is a mac-only issue. It uses NS to draw, not X11 etc.

[hraban]

I'm also getting flickering behaviour in Mac OS, only from nixpkgs.

I've started collecting various methods for building Emacs in Nix in a single flake: "https://github.com/hraban/emacs-nix". So far, all three methods give me the same flickering.

When I build Emacs locally, from source, without Nix at all, I don't get any flickering. Here's what I use:

CC=/usr/bin/clang make configure="--enable-link-time-optimization --with-imagemagick --with-json --with-modules --without-dbus CFLAGS='-O3 -march=native'"
make install

N.B.: This is pretty similar to my github:hraban/emacs-nix#from-src derivation, but of course it must be picking up on some kind of system-wide dependency that isn't available to the Nix build. Notably it's the exact same Emacs source (latest master), without any patches, so for sure this is a build problem.

The nix wiki's suggestion to use the community overlay and apply mac-port patches (github:hraban/emacs-nix#mac-port) doesn't solve it for me, either.

The closest description I've found of the symptoms I observe is in this obscure Stackoverflow answer: "https://emacs.stackexchange.com/a/60005". He seems to talk, though, about the Mac port---note that I observe the same flickering in a build from unmodified Emacs source without any patches.

And since I'm braindumping my entire Emacs+nix+mac story here, there's another odd symptom I observed with Nix builds: if I C-g at the "wrong" time, it looks like I completely kill some kind of "redraw" process, and Emacs completely stops drawing anything until I restart it. It accepts hotkeys fine (C-x C-c 🙄).

Sorry for the unstructured state dump. I'm still debugging this and I'll post here with further updates as I find them. But if I don't write this down now I'll never post :P

PS: I'm absolutely no Mac build specialist, but afaik MAC_OS_X_VERSION_MAX_ALLOWED is about the latest version of Mac that your source code can target. So you wouldn't break anything before anything, but any mac OS APIs that you rely on that were available after that version won't be available to your program. As I understand it, if you don't care about older macs but instead want to allow your app to use the latest and greatest APIs, you want to play with MAC_OS_X_VERSION_MIN_REQUIRED. Mac system headers have a table explaining this, which gets more confusing the longer I look at it.

Please send me any suggestions you have for derivations. I'm happy to add them to the flake and grow our motley corpus of broken Emacs builds. :)

[antifuchs]

PS: I'm absolutely no Mac build specialist, but afaik MAC_OS_X_VERSION_MAX_ALLOWED is about the latest version of Mac that your source code can target. So you wouldn't break anything before anything, but any mac OS APIs that you rely on that were available after that version won't be available to your program. As I understand it, if you don't care about older macs but instead want to allow your app to use the latest and greatest APIs, you want to play with MAC_OS_X_VERSION_MIN_REQUIRED. Mac system headers have a table explaining this, which gets more confusing the longer I look at it.

So I've been building my own emacsUnstable (from a current emacs-overlay) with CFLAGS = "-g -O3 -DMAC_OS_X_VERSION_MAX_ALLOWED=110203" for the past year or more, and haven't noticed a redraw issue anymore (definitely did before I started using the max_allowed definition). What you mention about max/min versions of the source code used to compile kinda makes sense to me? This is behavior that for most people started with newer versions of macOS (if I recall correctly! - please correct me if that's not the case). So if the drawing routines that emacs uses changed from, say Mavericks to Big Sur, it would make sense that opting into the older behavior via MAX_ALLOWED would restore the (nice) old behavior.

I think if that is true, the real fix will be to figure out what exactly changed between those releases (and which releases they were!) that emacs's redraws are now causing flickers - and fix that. But in the meantime, I think opting into the older code seems right?

[hraban]

PS: I'm absolutely no Mac build specialist, but afaik MAC_OS_X_VERSION_MAX_ALLOWED is about the latest version of Mac that your source code can target. So you wouldn't break anything before anything, but any mac OS APIs that you rely on that were available after that version won't be available to your program. As I understand it, if you don't care about older macs but instead want to allow your app to use the latest and greatest APIs, you want to play with MAC_OS_X_VERSION_MIN_REQUIRED. Mac system headers have a table explaining this, which gets more confusing the longer I look at it.

So I've been building my own emacsUnstable (from a current emacs-overlay) with CFLAGS = "-g -O3 -DMAC_OS_X_VERSION_MAX_ALLOWED=110203" for the past year or more, and haven't noticed a redraw issue anymore (definitely did before I started using the max_allowed definition). What you mention about max/min versions of the source code used to compile kinda makes sense to me? This is behavior that for most people started with newer versions of macOS (if I recall correctly! - please correct me if that's not the case). So if the drawing routines that emacs uses changed from, say Mavericks to Big Sur, it would make sense that opting into the older behavior via MAX_ALLOWED would restore the (nice) old behavior.

I think if that is true, the real fix will be to figure out what exactly changed between those releases (and which releases they were!) that emacs's redraws are now causing flickers - and fix that. But in the meantime, I think opting into the older code seems right?

Interesting, I see what you mean: maybe the problem is a new Mac OS API that Emacs is using, when available. And disabling that would prevent flickering.

I rebuilt with your suggested flag, but I still get flickering unfortunately. Maybe we have different root causes?