Animation loop for real time rendering

[erf]

I just had this idea it might be fun to hook up Wezterm to an animation loop (like a game engine) using the LUA API which would render the terminal in 60fps and give you the time since last render call (dt) so you could do custom rendering / animation for each background cell (or using a custom drawing layer) for this. You could perhaps also change the settings for each render call, like changing the gradient colors in real time or just setting each cell color or a subset of it. You could perhaps also add your own high res image on top of the terminal and change it's values over time.

The render loop would by default be turned off to save battery, but could be turned on untill stoppped, scheduled for later or scheduled to be turned off after a while.

Not sure if this sort of thing is possible right now or in the pipeline, but i could imagine it could open for some potenial fun visual experiences :)

[digitallyserviced]

So i figured I would reply to you. Just so you can have a taste maybe of what you could do for effects.

It is not really feasible to have a real time rendering at all really, especially with an interpeted language like lua....

However being that this is a hardware accelerated terminal that uses OpenGL... The ability of doing fragment shaders to generate pretty shit like (shadertoy.com) is not too difficult. It is however GLSL that is more of a math/shader language than programming specifically...

This was super proof of concept and unpolished and needs a lot of tweaking in places but I have spent too much time on putting this together then I should have... Whether I continue or take this further I am not sure,

This is not something I want to be demanded as a feature though because it can be a pain in the ass, but I have had a bit of experience with building and rendering shaders on android so it was fun to play with.

[erf]

That looks pretty cool. Custom shaders would be nice too. What i was thinking is to use Lua to give commands (draw per char background) to some lower level api which would render to a separate buffer / layer using the graphics api (shaders). This is used by many game engines, like Love2D, Solar2D etc.

[digitallyserviced]

I understand what you mean. The ability to use lua to influence shader uniforms(arguments/parameters) or update the shader programs may be something that can be implemented, however wezterm is a terminal emulator, and not really built as a graphics engine.

THe gif is short and may look jumpy/fidgety because it only renders a frame if there is a change to the terminal, or there is input being captured, mainly because there is no idea or need for a 'render loop'.

The other thing is that the engines you mention are made specifically with that idea in mind, that there are frames constantly being rendered, there are changes that occur each frame, or tick, or as response to user input. These are all things that while may be implemented in some way, are not built with the idea that they are continuously called with multiple events expected. Usually they are not rendered multithreaded, rather a single thread renders graphics.

Wezterm was built to be optimized and not utilize the CPU or graphics when no changes to the screen are being made.

While it is a nice thought and there are the abilities to add graphics effects to things, I do not see there ever really being anything close to what you originally asked for.

I am interested in investigating the shaders for backgrounds so may at least push my branch if people are interested in fucking with it.

[erf]

I imagine a render loop would be opt-in only if Mr Wez is in support of this idea.

[wez]

Some of what goes through my mind for feature requests is:

• Could most people use it?
• Could they use it without it being really tricky?
• What impact does it have on my ability to maintain/evolve wezterm?

The last one is the main thing I worry about when it comes to eg: allowing users to put stuff in shaders. If we allow full access to the shader program then it makes the shader language itself part of our API and may pose problems if, for example, I wanted to port directly to Metal on macOS, or Direct3D on Windows, which don't use exactly the same shader language as OpenGL.

TBH, shaders are complex and I don't expect most users will want to deal with them.

The way I like to look at this sort of request is to understand what kinds of features are we trying to unlock and then see if there is a sustainable way to enable them that doesn't lock wezterm into supporting something difficult in terms of implementation or difficult for users.

When it comes to rendering stuff from lua it makes me think of the existing overlay mechanism in wezterm: the debug, launcher, copy mode and other overlays are implemented as a special kind of TUI application that runs in-process. Those are all implemented in Rust but it would be feasible to expose the key functions to lua and enable lua-implemented overlays to be built.

While I referred to those as TUI programs, they build on top of the termwiz/wezterm Cell, Line, Change and Surface data model which is capable of representing image and even animated image attachments and is capable of providing pretty rich experiences.

If your goal is to do something fancy with background layers and gradients, say, using a radial gradient and changing its position and coloring to match the sun rising and falling, then the moon rising and falling as the time of day changes, then that doesn't fit into the overlay model and we'd need something else for it.

Thinking more simply about it: https://wezfurlong.org/wezterm/config/lua/config/window_background_gradient.html is deliberately very declarative in terms of what the user places in their config. Today the implementation for it is that wezterm generates an image based on that config and then tells the shader to render than texture across the background. We could instead expand the shader so that we can pass in the data from the config and teach the shader to directly render the gradient. We shouldn't need to change the config around that to facilitate that kind of implementation detail change.

Getting back to animating it: if we could declaratively express the animation then we have a certain amount of freedom in the implementation: we could pre-render the animation frames to textures and then change them at the appropriate time (that's basically how animated gif/png files work in wezterm today), or we could teach the shader how to evaluate the deliberately constrained rules.

Now, the question is: what is the maximum degree of complexity/flexibility needed for the use cases you have in mind? There's a point at which this sort of thing would essentially turn into a version of the shader language which probably means that we've gone too far!

I actually think that computing frames for animated pngs is probably not a bad thing to target: I know it doesn't allow for some of the fancy things that are possible with shaders, but if there were functions in wezterm that allowed composing those (eg: exposing the tinyskia crate that we use to render 2D vector graphics, as well as the image crate functions), and then make it so that you could pipe that into the background image layer stuff, or pass that to a Cell in an overlay app. I actually want something like this for:

• https://github.com/wez/wezterm/issues/1432

If what you're looking for instead is doing stuff like emitting pixel-perfect gradients as the text background for a span of text then that's the sort of thing where I thing adding a new SGR escape sequence to represent it in the terminal data model would make sense, and then we can explicitly teach the renderer how to present that. That's not really end-user-customizable, but it is much more sustainable.

So, this is a really long winded way of asking: what exactly are you hoping to do with this feature, so that we can find a way to constrain the scope and make it more sustainable to support and use?

[erf]

Thanks for the thoughtful answer.

What i was thinking was to hook up to a animation event loop with given time intervals (or schedule ahead of time) in the config file so i could make little programs which would change the appearance of the background. Like the sunset you mention or make some fun color animations per char.

I'm thinking a common high level Lua API to change:

• each cell / char color
• image background layer (per pixel)
• window background radients
• other config settings ..?

It sounds like pre rendering images is not exactly what i'm looking for but rather be able change a image buffer or cell color for each render event.

I'm not too concerned with shader support, but rather a high level Lua api.

[digitallyserviced]

I am kinda with @wez when it comes to adding some of what you mentioned...

• background gradients and the background already are rather customizable, albeit not animated (without parallaxing).
• this ultimately is a terminal, and not a gfx canvas... This is a vt100 canvas and using things like Textual or how btop draws its stats display. It makes sense to support features for the terminal emulation, not for canvas drawing...
• Each cell/char color I ultimately do not understand. I do not see why you would want a separate color/attribute definition that is separate from what the tty/tui produces itself.
• background/animations in any form/fashion... The biggest issue with a background or animation, or transparency is the contrast between it and the text in the console. Many times I have had to increase opacity or darken backgrounds so that the color scheme does not have issues with text not able to be seen.

Most of what I see possible would be features alraedy existing in wezterm but now just tying those into a user accessible lua/cli API. THe most powerful part of wezterm is the lua API, and usually as long as the feature works with wezterm's own functions, the lua API works without a hitch, and since it is the user's introduced code, it's on them to resolve/fix.

• Increase wezterm's lua api surface area. More hooks/functions to control more fine details like pane content, sizes, overlays, modals.
• Allow modals/overlays to have content defined via a lua func/coroutine/hook. This way panes/modal/overlay could be used similar to tmux where you could pipe a commands output, or even display your own text to it, and/or receive input/interactivity.
• If we were to allow shaders to be customized then it would be quite simple and all issues would then fall to the user since they're code is now being introduced.
• Offer pre-made shaders/gfx features that could have an effect on the cursor or visible activity bell's, or on pane/window resizes.

While I sort of understand what you are asking for with the lua rendering stuff... I really do not 100% understand it. Only because I see the terminal as something that needs to be responsive, be consistent and predictable, clean and simple to scan the output/content, and ultimately not crash.

[erf]

Here is some inspiration for some really interactive text editing done by tscoding in his Dramatic Text editor :)

https://www.youtube.com/watch?v=V-o1cKaWOQc&t=100s

Not sure if it is really something you'd want in the terminal, but it kind of show you the possibility if it was structured as a game / render engine. Moving the camera around could be done via the Lua API by listening to cursor change events, etc.

[erf]

I'll close this for now as this was perhaps too out there for TE functinality

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.