Animate window move/resize/swap/warp

[dominiklohmann]

Discussion / Feature request

This is a long shot, and probably far off in the future. This has been previously discussed in https://github.com/koekeishiya/chunkwm/issues/144.

I've dug up some stuff regarding this.

• How Chromium resizes and moves windows using CGS* functions https://github.com/ChromiumWebApps/chromium/blob/c7361d39be8abd1574e6ce8957c8dbddd4c6ccf7/chrome/browser/ui/cocoa/constrained_window/constrained_window_animation.mm
• A (very old) blog post explaining how CGSSetWindowWarp works http://kevin.sb.org/2006/07/23/cgssetwindowwarp-explained/
• An working example for using CGSSetWindowWarp and CGSSetWindowTranslation for custom animations https://github.com/sailesha/CGSSetWindowWarp-Sample/blob/master/windowAnimation.mm

Obviously all the above examples require access to the NSWindow instance for async animations, which is inaccessible from the Dock process—which is the only process yabai injects code into.

Simply calling SLSSetWindowTransformation with a CGAffineTransform (e..g CGAffineTransformMakeTranslation(100, 100) to move a window 100 right and 100 down) will move the window, but this is not animated. Calling this repeatedly will likely cause lag.

And now here's the long shot that I'd love to see investigated in the future: There are additional functions called SLSTransactionCreate and SLSTransactionCommit, with variants of the transform and warp functions named SLSTransactionWindowTransform and so on. Assuming this works similar to CATransaction on iOS, this could then be used to animate multiple windows with a duration and at the same time.

These SLSTransaction* functions are available (on 10.15 dev beta 4 currently):

nm /System/Library/PrivateFrameworks/SkyLight.framework/SkyLight | grep SLSTransaction

``` 0000000000200d1e T _SLSTransactionAddWindowToSpace 0000000000200f3a T _SLSTransactionAddWindowToSpaceAndRemoveFromSpaces 0000000000201f5c T _SLSTransactionBindSurface 0000000000200129 T _SLSTransactionClearMenuBarSystemOverrideAlphas 00000000001ffce4 T _SLSTransactionClearWindowLockedBounds 00000000001ff834 T _SLSTransactionClearWindowSystemLevel 0000000000202cb6 T _SLSTransactionCommit 0000000000202ee0 T _SLSTransactionCommitCoalescing 00000000001fd88b T _SLSTransactionCreate 0000000000201a22 T _SLSTransactionDeferWindowMoveEvents 00000000002013c1 T _SLSTransactionDestroySpace 00000000001fd788 T _SLSTransactionGetTypeID 000000000031b9d0 b _SLSTransactionGetTypeID.once 000000000031b9d8 b _SLSTransactionGetTypeID.typeID 000000000020109f T _SLSTransactionHideSpace 00000000001ffd7d T _SLSTransactionMoveWindowWithGroup 0000000000201457 T _SLSTransactionMoveWindowsToManagedSpace 000000000020205f T _SLSTransactionOrderSurface 00000000001fe097 T _SLSTransactionOrderWindow 00000000001fe196 T _SLSTransactionOrderWindowGroup 0000000000201c0a T _SLSTransactionOverrideAppSleepNotifications 0000000000201eb2 T _SLSTransactionPostBroadcastNotification 0000000000202dd4 t _SLSTransactionPostCommit 0000000000201d2a T _SLSTransactionPostNotificationToConnection 0000000000200dd1 T _SLSTransactionRemoveWindowFromSpace 0000000000200e84 T _SLSTransactionRemoveWindowFromSpaces 000000000020132b T _SLSTransactionResetSpaceMenuBar 00000000001fff5c T _SLSTransactionResetWindow 00000000001fef10 T _SLSTransactionResetWindowSubLevel 0000000000202922 T _SLSTransactionSetBackdropChameleonContribution 000000000020283c T _SLSTransactionSetChameleonUpdatesEnabled 0000000000201ec6 T _SLSTransactionSetClientAdvisory 0000000000201950 T _SLSTransactionSetEventCapture 0000000000201654 T _SLSTransactionSetManagedDisplayCurrentSpace 0000000000201875 T _SLSTransactionSetManagedDisplayIsAnimating 00000000002001a9 T _SLSTransactionSetMenuBarBounds 00000000001ffff5 T _SLSTransactionSetMenuBarSystemOverrideAlpha 0000000000200a84 T _SLSTransactionSetSpaceAbsoluteLevel 0000000000200bea T _SLSTransactionSetSpaceAlpha 0000000000200b37 T _SLSTransactionSetSpaceOrderingWeight 0000000000200957 T _SLSTransactionSetSpaceShape 000000000020054f T _SLSTransactionSetSpaceTransform 0000000000202175 T _SLSTransactionSetSurfaceBounds 0000000000202563 T _SLSTransactionSetSurfaceOpacity 0000000000202666 T _SLSTransactionSetSurfaceResolution 00000000001fe295 T _SLSTransactionSetWindowAlpha 00000000001fe3c6 T _SLSTransactionSetWindowBrightness 00000000001ff8cd T _SLSTransactionSetWindowGlobalClipShape 00000000001fe4f7 T _SLSTransactionSetWindowLevel 00000000001ff9fd T _SLSTransactionSetWindowLockedBounds 00000000001fe66b T _SLSTransactionSetWindowOpaqueShape 00000000001fe79b T _SLSTransactionSetWindowProperty 00000000001fefa9 T _SLSTransactionSetWindowReleasesBackingOnOrderOut 00000000001feace T _SLSTransactionSetWindowResolution 00000000001fed2a T _SLSTransactionSetWindowShape 00000000001fee5a T _SLSTransactionSetWindowSubLevel 00000000001ff08f T _SLSTransactionSetWindowSystemAlpha 00000000001ff77e T _SLSTransactionSetWindowSystemLevel 00000000001ff1c0 T _SLSTransactionSetWindowTransform 00000000001ff5e9 T _SLSTransactionSetWindowWarp 0000000000201009 T _SLSTransactionShowSpace 000000000020158b T _SLSTransactionSpaceTileMoveToSpaceAtIndex 0000000000201af2 T _SLSTransactionUpdateRegion 000000000020120c T _SLSTransactionWillSwitchSpaces 00000000001fe615 t __SLSTransactionCommitAction 00000000001fdfe4 t __SLSTransactionDecode_CFString 0000000000052ef0 t __SLSTransactionDecode_CGSSpaceIDArray 00000000001fdc6b t __SLSTransactionDecode_packed64 00000000001fdc9e t __SLSTransactionDecode_packed64_tag 0000000000201135 t __SLSTransactionEncode_CFArray 00000000001fdd41 t __SLSTransactionEncode_CFString 00000000002016fd t __SLSTransactionEncode_UUIDString 00000000001fdba3 t __SLSTransactionEncode_packed64 00000000001fd994 t __SLSTransactionEncode_resize 00000000001fd7cd t __SLSTransactionFinalize 00000000001fd925 t __SLSTransactionPopWriteStream 0000000000201d3a t __SLSTransactionPostNotification 0000000000202a53 t __SLSTransactionPushWriteStream 0000000000202bf6 t __SLSTransactionResetWriteStream 000000000031b9e8 b __SLSTransactionWriteStreamKey.key 000000000031b9e0 b __SLSTransactionWriteStreamKey.once 000000000020300f t ___SLSTransactionCommitCoalescing_block_invoke 00000000001fd7b4 t ___SLSTransactionGetTypeID_block_invoke 0000000000201507 t ___SLSTransactionMoveWindowsToManagedSpace_block_invoke 0000000000202ed8 t ___SLSTransactionPerformCommitAction 0000000000202520 t ___SLSTransactionSetSurfaceBounds_block_invoke 0000000000202808 t ___SLSTransactionSetSurfaceResolution_block_invoke 00000000001fe5ef t ___SLSTransactionSetWindowLevel_block_invoke 00000000001feca3 t ___SLSTransactionSetWindowResolution_block_invoke 00000000002012a5 t ___SLSTransactionWillSwitchSpaces_block_invoke 00000000001fdad3 t ____SLSTransactionWriteStreamKey_block_invoke ```

Why? Look at this example from i3. Looks nice, doesn't it?

[koekeishiya]

I'm not particularly interested in doing this, but would not be against a PR where this is optional.

[danielfalbo]

Another animation idea: it would be really cool to animate the opening of the terminal in a sci-fi style, kinda like edex-ui . Of course it would be cool for other windows too but especially for the terminal because it's usually really flexible in terms of size and is the app that feels the most like a sci-fi thing 🤪.

Alacritty, for example, is completely resizable. In the gif I put an Alacritty window in a floating space, and, after trying to make it as small and centered as possible, I first do yabai -m window --resize left:-20:0 and yabai -m window --resize right:20:0 multiple times until the window horizontally fills the screen, and then the same thing vertically with yabai -m window --resize bottom:0:20 and yabai -m window --resize top:0:-20

[ghost]

Any update on this?

[albert-ying]

Just saw this on reddit, it looks so smooth

https://www.reddit.com/r/unixporn/comments/vtpwhc/hyprland_dynamic_wallpapers_and_settings_with_a/if9iaoy/?context=3

[koekeishiya]

Just to note that the mentioned SLSTransaction function family does not work in Monterey 12.5. (Maybe there are entitlements or whatever you can somehow add to your binary to make them work, but they no longer work OOTB as they do in Big Sur).

[itaysharir]

Did anyone manage to implement it and has a compiled version of it?

[ghost]

im wondering of this too

[FelixKratz]

https://user-images.githubusercontent.com/22680421/189494638-f8a63c68-297b-41d0-9805-c5cf0f02b33a.mp4

I figured it is not a big problem to call SLSSetWindowTransformation frequently to create the intermediate frames. I have hardcoded there to be 100 frames in any resize/move animation.

[ghost]

brilliant.

anim.mp4

I figured it is not a big problem to call SLSSetWindowTransformation frequently to create the intermediate frames. I have hardcoded there to be 100 frames in any resize/move animation.

brilliant.

[koekeishiya]

@FelixKratz

Did you only apply the transform in that POC, or does it also synchronise said transform with the desired window frame change? This is important because macOS itself applies transforms to windows in various contexts, and that will undo the operation.

[FelixKratz]

@FelixKratz

Did you only apply the transform in that POC, or does it also synchronise said transform with the desired window frame change? This is important because macOS itself applies transforms to windows in various contexts, and that will undo the operation.

It is not polished at all and very hacked together, this is all I did: https://github.com/FelixKratz/yabai/commit/2ba29b73df0c9e71cd3d34bf0c97ef2e59cfa7bf I am only setting the transform for as long as I am animating and then I do the actual proper move/resize. These animations block yabai, which is why I have made them so short.

[koekeishiya]

I'll probably take a shot at this for the next release, as I'm working on smoothing out the experience. I have a fairly high bar for what I consider acceptable though, so not sure if it is feasible to ship in the end. A dealbreaker for me is the ability to animate all windows that are modified in the same operation simultaneously, so that they start/end at the same time.

[FelixKratz]

I'll probably take a shot at this for the next release, as I'm working on smoothing out the experience. I have a fairly high bar for what I consider acceptable though, so not sure if it is feasible to ship in the end. A dealbreaker for me is the ability to animate all windows that are modified in the same operation simultaneously, so that they start/end at the same time.

Great to hear! Animating the windows in parallel would be very nice, but quite a bit of restructuring of the code base I believe. It would probably also be much better to animate from the SA directly. I only chose to not do that because then the SA would have to notify yabai when the animation is finished or unlock some kind of mutex that the SA and yabai share and that would go too far for a simple POC. This would probably also benefit greatly from your new ipc approach via shmem + mutex. When those animations are done "proper" (parallel and less flickering) it will feel very nice I am sure.

What I stumbled over while implementing the POC was that once the affine transform is reset to the original transform before moving and resizing the window via AX there would sometimes be a noticeable flickering and I didn't find a way to resize the window through SLS functions, i.e. SLSSetWindowShape does not seem to be respected when set via SA. So I was stuck with the slow resize via AX.

[koekeishiya]

Pushed a proof of concept myself: https://github.com/koekeishiya/yabai/tree/animate https://github.com/koekeishiya/yabai/commit/269d29d2ddbdd22c4024a0151f461a3dd9def616

Remember to reinstall the scripting addition after building the above branch. Add yabai -m config window_animation_duration 0.35 to enable animations; adjust the value as you like. This does not work with window borders yet, so keep borders disabled.

This still only animates a single window at a time.
Multiple windows are animated simultaneously: https://github.com/koekeishiya/yabai/commit/45e18e0d7b65b4dab403f79986c7558ee1ea8f2f

What I stumbled over while implementing the POC was that once the affine transform is reset to the original transform before moving and resizing the window via AX there would sometimes be a noticeable flickering and I didn't find a way to resize the window through SLS functions, i.e. SLSSetWindowShape does not seem to be respected when set via SA. So I was stuck with the slow resize via AX.

I don't think it is possible to get around this; I have attempted to do so in my version, but no guarantees that it is a universal solution.

[koekeishiya]

Transactional:

sample showing repeated calls to yabai -m space --rotate 90.

The flicker is caused by the issue mentioned above; when resetting the transform at the end of the animation to apply the change in size using the AX API. Hopefully there will be a workaround to that...

[koekeishiya]

Resizing using the AX API is just too slow. Need to figure out some other way to resize a window.. When resizing a window using the native (mouse) handle at the edges, the operation appears to be rather snappy. I wonder if we could somehow hook into that.

[FelixKratz]

When looking at the animation happening when clicking the green fullscreen button on any windows handle I got a (janky?) idea:

• Spawn a new window showing a capture of the surface of the window to be animated
• Position the "fake" window exactly where the "real" window is located
• Make the "real" window invisible (alpha -> 0)
• Resize and reposition the "real" window (while invisible)
• Animate the transition via the "fake" window
• Make the "real" window visible again (at correct position and size already)
• Remove the "fake" window

I don't know if this is ok to do performance wise, but I suspect something similar is happening in the system animation to fullscreen. They additionally transition between the scaled versions of the unresized and the resized window through the animation, making it look a bit smoother.

[koekeishiya]

I had an idea that it would maybe be ok to animate the actual windows as is done, but taking a screen capture of the space at the final frame; displaying this screen capture as a topmost window and destroy it (probably some kind of blurred fade) while removing the transforms and applying the resize animation.

Thanks to your comment now I recorded my screen while doing the enter/exit fullscreen application, and it actually looks like Apple themselves are doing something like this, surprisingly enough.

[FelixKratz]

I tried my idea by misusing the border to temporarily draw the capture of the window. The scripting addition is not needed for the animation at all (only for changing the opacity) and it looks very smooth. https://github.com/FelixKratz/yabai/commit/18eb2ca9d2caf1fffb8097e0378efd9826ff54c7

[koekeishiya]

Are you able to animate the windows simultaneously using that approach though?

Edit:

I guess you could iterate through all windows, grab a capture of the window, and set the opacity to 0. Set the size and position using the AX API.

Animate all the captures.

Iterate through the list of windows, setting the opacity back to the previous value.

[FelixKratz]

Are you able to animate the windows simultaneously using that approach though?

Edit:

I guess you could iterate through all windows, grab a capture of the window, and set the opacity to 0. Set the size and position using the AX API.

Animate all the captures.

Iterate through the list of windows, setting the opacity back to the previous value.

In my current implementation it is single window, but if I pass a window list to the animation function (same as you have done) it will be possible to animate simultaneously. Maybe I will try to rebase to your POC and implement this method there, but I think the digest of this is that we can definitely circumvent the flickering. Maybe it would be good to use the faster SLSHW functions for window capture instead of what I am using currently.

[koekeishiya]

It works pretty well, except that the window order is not the same as it is pre-screenshot, and sometimes the screenshot colours look different from the window itself. e.g Safari is darker than it is in the screenshot. This also breaks the window fading option if an animation is played on window create/destroy/focus change.

Edit: So the issue appears to be that the screenshot does not capture vibrancy effects among other things

[koekeishiya]

This is probably as good as it is going to get: https://github.com/koekeishiya/yabai/commit/4136ac99888eafefbf02dc95754632c572854a3c

There are some artifacts in the gif due to compression and whatever, it looks much better in reality.

Slight issues with border integration still, but the animation system is pretty much done.

[FelixKratz]

I have rebased onto your POC and played with it for a bit. I have made some slight tweaks that get rid of the border flicker and reduce the overhead if borders are activated: https://github.com/FelixKratz/yabai/commit/681abdceee0be433823cb4b7c5696088b0a55220

[koekeishiya]

@FelixKratz

Does that solve the border misalignment when using the mouse to move/resize windows (especially noticable when doing a warp operation using the mouse) ? I haven't looked to closely at the border stuff yet.

Does anyone have a set of functions that look nice for animating various kinds of window movement. Might be nice to allow the ability to select between a variety of functions, and possibly select an animation based on which action is happening.

[FelixKratz]

@FelixKratz

Does that solve the border misalignment when using the mouse to move/resize windows (especially noticable when doing a warp operation using the mouse) ? I haven't looked to closely at the border stuff yet.

No, I did not notice it until now since I rarely use my mouse for those actions. But the animation system so far is very nice and helps to indicate direction of movement, especially for other people viewing the screen.

(The change in the transforms is only to accommodate for my rounded border below the window with an inset respective to the windows frame)

[FelixKratz]

Does anyone have a set of functions that look nice for animating various kinds of window movement. Might be nice to allow the ability to select between a variety of functions, and possibly select an animation based on which action is happening.

• $f(x) = x$
• $f(x) = x^2$
• $f(x) = \sin(x \pi / 2)$
• $f(x) = 1 - \cos(x \pi / 2)$
• $f(x) = x \exp(x - 1)$
• $f(x) = c \tanh(2 \text{artanh}(\frac{1}{2c})(x - 1/2)) + 1/2$, for $c \in (1/2, \infty)$

all of those functions enforce $f(0) = 0$ and $f(1) = 1$.

I am not sure if they are "nice" but they are smooth and infinitely differentiable on the domain. I especially like the last one for $c \sim 0.52$

[Liquidmantis]

I don't have anything to add to the discussion, so apologies for the noise, but I just wanted to say thanks to you two for all the effort on this. It looks amazing and I can't wait.

[koekeishiya]

I think I found a way for this entire thing to work without having to disable SIP. Will try to implement it in the coming days.

Edit: Animations have become more complicated since this comment was written. The workaround doesn't work with the updated requirements, so animations will require SIP to be disabled.

[FelixKratz]

I wonder if it would be possible to make these animations not block yabai, similar to how I handle animations in SketchyBar: https://github.com/FelixKratz/SketchyBar/blob/2008f71c2162c8c4d16f007ebc9f798d8457ba15/src/animation.c#L157-L177 The implication of this would be that something like:

yabai -m window --toggle zoom-fullscreen \
&& yabai -m query --windows --window | jq '."has-fullscreen-zoom"'

would need to return true when the animation has begun and not only after it has ended.

[koekeishiya]

It is possible to make them non-blocking, yes. However, then we need to handle the case of the user performing some action on a window that is being animated and correctly cancel the animation for that specific window or something, similar to what I did for the window opacity fading system in #1406

In addition, the queued events will then also be processed while a window is animating, e.g we will receive and process a window resize event for the windows that are animated, while they are being animated, causing their cached properties to update (and it will also trigger a border update).

I'm not sure there is too much to gain by making them run asynchronously.

[FelixKratz]

I'm not sure there is too much to gain by making them run asynchronously.

It could make some things feel a bit better, take for example:

ctrl + lalt - j    : yabai -m window --resize right:-100:0 || yabai -m window --resize left:-100:0
ctrl + lalt - k    : yabai -m window --resize bottom:0:100 || yabai -m window --resize top:0:100
ctrl + lalt - l    : yabai -m window --resize bottom:0:-100 || yabai -m window --resize top:0:-100
ctrl + lalt - 0x29 : yabai -m window --resize right:100:0 || yabai -m window --resize left:100:0

If I hold any of those key combos for a while I am in for a bad time, depending on my key repeat frequency this will create a resize queue several seconds long.

[koekeishiya]

If I hold any of those key combos for a while I am in for a bad time, depending on my key repeat frequency this will create a resize queue several seconds long.

What is the expected behaviour though, cancelling the current animation or simply queue up the next one?

I think both scenarios will look pretty janky. I guess it could look smooth by cancelling the current animation and have the new animation start interpolating from the position of the cancelled animation and not the final position (the AX Position of the frame).

This will break my "animation with SIP Enabled" workaround, but I am open for it if if makes the experience significantly better.

[FelixKratz]

What is the expected behaviour though, cancelling the current animation or simply queue up the next one?

At least for me the animation is a way to indicate the spacial directionality of a state change. The state change happens as soon as I request it, i.e. the state has already changed when the animation is started. Thus the expected behavior for me is to cancel the current animation and animate to the new state without stopping at the outdated states, because they have been invalidated by my new request.

Of course this is a matter of taste, the way it is currently the animation needs to finish for the state to change, but this leads to the feeling of not being responsive e.g. in the above scenario.

This also ties in with the other above command:

yabai -m window --toggle zoom-fullscreen \
&& yabai -m query --windows --window | jq '."has-fullscreen-zoom"'

For me it would be expected that yabai directly returns true and not wait till the animation is finished to return true.

Another example:

yabai -m window --warp south

almost instantaneously followed by:

 yabai -m window --warp west

should return the window west as soon as I request it. Especially with keycombos this can happen frequently while in an animation and make yabai feel sluggish (especially for long animation durations e.g > 1).

[koekeishiya]

I agree that makes sense. I think that would require changes to allow the following which I mentioned above:

I guess it could look smooth by cancelling the current animation and have the new animation start interpolating from the position of the cancelled animation.

[koekeishiya]

async animations poc:

Edit: The gif looks laggy, but it is actually really smooth.

[ghost]

can window open and close animations be added?

[koekeishiya]

Why is it so hard to make a decent gif..

I think this looks better maybe. Changed the easing function and removed weird flickering that was present in the previous version:

[FelixKratz]

Wow, it works really well and it is exactly how I would expect an animation system to work! Thank you for making this, very nice work!

[koekeishiya]

Squashed all changes so far into: https://github.com/koekeishiya/yabai/commit/e33fc6f90d5ba709d98cf6c9f49ed9a6f5c4c020

[koekeishiya]

can window open and close animations be added?

It might be possible to create a window opening animation; depends on how much delay there is between the window spawning and yabai getting notified, I guess this would likely be implemented as a fade+scale transition.

Animating on window close is pretty much impossible. The window is already inaccessible from system APIs by the time we are notified that a window has been destroyed.

[koekeishiya]

Window borders will be hidden while windows are animating. Attempting to do otherwise will add a ton of complexity and prone to use-after-free issues.

[FelixKratz]

The EVENT_HANDLER_SLS_WINDOW_RESIZED functions are called several times during an animation, leading to a resize of the border and inflicting some lag with large resolution windows. It would probably be best to guard against a border resize during an animation?

EDIT: I think the problem is that updates are disabled during resizing of the border, leading to dropped frames in the animation, but I could be wrong.

[koekeishiya]

The EVENT_HANDLER_SLS_WINDOW_RESIZED functions are called several times during an animation, leading to an unnecessary resize of the border and inflicting some lag with large resolution windows. It would probably be best to guard against a border resize during an animation?

There should only be one call to that event handler for each set_window_frame call that causes a window to change size. The current system basically does the following:

Figure out which windows are involved in the animation. Grab a snapshot for each window and prepare inital state. Hide the real window and its border; show the proxy. Set the window frame of the real window (this triggers the callback that the border responds to, events are currently queued). Start the animation in a background thread. Queued events continue being processed by yabai as normal.

When an animation ends, it hides the proxy and shows the real window, triggering the callback to display the border window again.

[FelixKratz]

I have tested my suspicion from above and it seems to be indeed caused by the disabled update: https://github.com/FelixKratz/yabai/commit/a58ce7a083bbf1d5f3cbc774c44397408a41270d on border resize

[koekeishiya]

I might have an idea to allow borders to animate; will look into it later.

[koekeishiya]

My idea appears to work; borders are animating fine, but needs some cleanup.

[koekeishiya]

Trying to animate the border directly causes flicker during the end of transition, so the best solution would simply be to create a proxy for the window border too, and pass them through the same system that we do for the windows themselves.

[koekeishiya]

I think this is getting close to finished now. I squashed all changes into 2 commits on the animate branch. It would be nice if someone else besides me could run this for some time and report unexpected behaviour across the entire feature set of yabai.

[FelixKratz]

I think this is getting close to finished now. I squashed all changes into 2 commits on the animate branch. It would be nice if someone else besides me could run this for some time and report unexpected behaviour across the entire feature set of yabai.

There still was some flickering occasionally, related to the reordering of windows. With the following changes the flickering is eliminated: https://github.com/FelixKratz/yabai/commit/08d7314b9d4051856846e6033f19dd6e4101c1bd