Strong flickering showing animations / movies with iterm2 image sequences.

[hzeller]

What Operating System(s) are you seeing this problem on?

Linux X11

Which Wayland compositor or X11 Window manager(s) are you using?

KWin from KDE desktop environment.

WezTerm version

wezterm c1f495099ef544eda78546191c0026e88e1fdd62

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version of the nightly that I tried

Describe the bug

There is a strong flickering when showing animated images or movies with timg (version 1.4.5).

Timg sends a sequence of images, places the cursor up to the beginning and send the next image. The image is emitted as PNG.

It looks a bit like the screen is blanked while the image is still being decoded, which leaves a brief blank period ?

Curiously, in a repeated animation, when it repeats, it runs a lot smother. Even though the images are sent continuously maybe there is a content-addressed cache ?

(Note: this is built from head. Same with nightly App-image The 20230408-112425 release does not show anything, so that is broken)

To Reproduce

If you have a video file or animated gif with some largish size (say 800x600), this should show manifest

Get timg from https://timg.sh/ or it should typically also be available on many distributions

timg some-movie-file.mov

Configuration

no config.

Expected Behavior

Smooth image sequence display.

Logs

wezterm version: c1f495099ef544eda78546191c0026e88e1fdd62 x86_64-unknown-linux-gnu
Window Environment: X11 KWin
WebGPU: name=Intel(R) Graphics (ADL GT2), device_type=IntegratedGpu, backend=Vulkan, driver=Intel open-source Mesa driver, driver_info=Mesa 23.1.1, vendor=32902, device=18086

^ this is my laptop machine.

This bug has the same behavior on my desktop with NVIDIA RTX 2080

wezterm version: c1f495099ef544eda78546191c0026e88e1fdd62 x86_64-unknown-linux-gnu
Window Environment: X11 KWin
WebGPU: name=NVIDIA GeForce RTX 2080, device_type=DiscreteGpu, backend=Vulkan, driver=NVIDIA, driver_info=535.54.03, vendor=4318, device=7810

Anything else?

This used to work :)

[hzeller]

This is how it looks like (Content warning: flickering) https://youtu.be/nX92cl91dec

[hzeller]

Interestingly, this only happens when sending the iterm2 protocol. When sending the kitty protocol (that wezterm understands as well in the meantime since WIP on #986), things work well.

The output can be chosen with the flag -pi (iterm pixelation) and -pk (kitty pixelation). If no option is given, timg recognizes wezterm and emits iterm protocol:

iterm2 image protocol

timg -pi some-video.mp4   # this is outputting a stuttering video

kitty protocol:

timg -pk some-video.mp4   # this output is smooth.

So this narrows it down to 'something broke in the processing of iterm2-image protocol images'.

[hzeller]

I could work around in timg to prefer kitty to iterm protocol when detecting wezterm.

Is kitty protocol preferred by wezterm these days ?

[wez]

The underlying cause of this is that wezterm recently moved arbitrary image decoding via the iterm2 protocol off the main thread to a background thread. This was to improve performance when decoding large images or animations, which could eagerly decode many frames at once and take a non-trivial amount of time.

The first frame decoding no longer blocks the main thread, and a blank texture may be shown if the renderer wins the race with the decoder.

When using the kitty protocol, decoding does happen on the main thread, so there is no blank visible.

Because timg is sending multiple iterm2 images, rather than say, an animated png or gif, it trips over the blank initial frame more often.

I'd like to eliminate the blank frame, perhaps by introducing some artificial small latency for the initial frame wait in the renderer, but care needs to be taken to avoid making it laggy.

re: which image protocol? The iterm2 protocol is less complex and more widely tested with wezterm; there are a number of kitty protocol bugs that are open against wezterm right now and you already hit one of them.

[wez]

I believe this to be resolved now in main; please give it a try and let me know!

[hzeller]

Ah, interesting, it sounds like something that might be fixable with some adapted synchronization where the renderer only sees things once they are in fact decoded. I am still in the early stages of learning Rust, but once getting more proficient I'll try to see how it is solved in wezterm.

w.r.t. timing, when you do the initial delay make sure to record the arrival time of the images (at least the ones that are written over the same location), and once emitted (which is some decoding latency apart) make sure that the next frame arrives at the renderer at the relative time distance when they have been read. That will make sure that animations run smooth and don't 'hickup', where two three images are emitted quickly and then there might be a pause.

(I do something similar in the reverse inside timg as compression is expensive; there I have futures that are filled by various threads but then picked out in sequence from the thing writing it to the terminal. Timings are conveyed relatively, so that after latency introduced by the encoding pipeline, animations run smooth: Each start of an animation marks time zero and then each subsequent frame contains the relative difference to the first frame. So once the emitter sees the first frame it remembers the time that happens and emits following frames with the right frame rate).

timg can't send apng in the general case as it can play arbitrary long videos which would probably require a lot of buffering, but I might consider it in the future for things with limited frames. If i do, would wezterm be able to receive a stream, i.e. be able to decode while I am still encoding a anpng ? That would be fantastic as it can reduce latencies.

Due to the flicker issue, I am using the Kitty protocol by default right now with wezterm, but am happy to switch back to iterm2 which is simpler and more robust.

[wez]

make sure that the next frame arrives at the renderer at the relative time distance when they have been read.

For animated image formats, wezterm will wait for the inter-frame delay specified by the image, rounded up to the minimum frame delay (that is, the time interval between frames based on the max_fps setting in wezterm: 1000ms / max_fps)

For non-animated images, like those used by timg, wezterm doesn't know that they are part of an animated sequence, so it has no idea on inter-frame timing; it will act on the incoming data as fast as it is received and processed. I think introducing some understanding of animations in this case will be quite complex and challenging, so I'm hoping that we can simply not do it!

timg can't send apng in the general case as it can play arbitrary long videos which would probably require a lot of buffering, but I might consider it in the future for things with limited frames. If i do, would wezterm be able to receive a stream, i.e. be able to decode while I am still encoding a anpng ?

The iterm2 image protocol doesn't support streaming, and even if it did, or we introduced another way to encode such a thing, the rest of the plumbing for managing image data makes use of content-addressed caches for sharing data and frames, so there would be some amount of effort required to support it. I'm not sure if the underlying image decoding crate used by wezterm would allow streaming decoding, which would be a blocker!

Maybe there is a hybrid approach: timg could potentially encode a few frames at a time and send a sequence of apngs? The heuristics would need to be good; eg: timg would need to successfully decide whether the encoding overhead to latency is worthwhile. Presumably there could be a saving in the encoded data size by doing that, so if small batches of frames work out cheaper (by some combination of size, latency, CPU overhead or other metrics) then it might be worth it. I don't know if a human would notice the difference unless the inter-frame intervals in the source animation are small enough.

[hzeller]

w.r.t. recognozing if something is part of an animated sequence: the animated frames are always starting at the same cell-position and have the same size, maybe that could be used as a hint that here is something that should be considered an animation ?

I'll play a bit with the partial apng-encoding idea, sounds like a good compromise. What would be the smallest apng, something with 2 frames, maybe even one frame ? At least that could then contain the frame timing information.

I tried to compile the latest version, but it immediately crashes with a segfault, looks like something inside some x11 keybinding thing. Will have to figure that out first.

``` #0 0x00007ffff756f31d in __strlen_avx2 () from glibc-2.37-8/lib/libc.so.6 #1 0x0000555556cd6723 in ::next () #2 0x0000555556b88825 in window::os::x11::keyboard::Keyboard::new () #3 0x0000555556b5d571 in window::os::x11::connection::XConnection::create_new () #4 0x0000555556c031b0 in window::os::x_and_wayland::Connection::create_new () #5 0x0000555555ec0500 in window::connection::ConnectionOps::init () #6 0x0000555555e4d2d2 in wezterm_gui::frontend::try_new () #7 0x0000555555d68b95 in wezterm_gui::run_terminal_gui () #8 0x0000555555d69e66 in wezterm_gui::main () #9 0x0000555555c85aa3 in std::sys_common::backtrace::__rust_begin_short_backtrace () #10 0x0000555555cbe089 in _ZN3std2rt10lang_start28_$u7b$$u7b$closure$u7d$$u7d$17hadb0e9ac78448dd0E.llvm.7425552176345397086 () #11 0x0000555557938911 in std::panicking::try () #12 0x00005555579180bb in std::rt::lang_start_internal () #13 0x0000555555d72445 in main () ```

[wez]

Hmm, I think that crash might be related to some debug diagnostics that I recently added. I just pushed a commit to remove those.

[wez]

w.r.t. recognozing if something is part of an animated sequence: the animated frames are always starting at the same cell-position and have the same size, maybe that could be used as a hint that here is something that should be considered an animation ?

The challenge is really that it is hard to reason about that without decoding the image at that stage, which we need to avoid for performance (and perhaps also future sandboxing/security) reasons, and it is difficult for wezterm to know whether an updated image is part of an animation sequence with implied timing vs. the user just showing a different static image of the same dimensions without any explicit information about it being an animation.

I'll play a bit with the partial apng-encoding idea, sounds like a good compromise. What would be the smallest apng, something with 2 frames, maybe even one frame ? At least that could then contain the frame timing information.

2 frames is probably the smallest that makes sense for animation, but I suspect that a few more than that would be more likely to be in the sweet spot of balancing encoding/decoding overheads. I think that sweet spot is likely going to be dependent upon the pixel dimensions of the frames, and depending on the encoder, the variations between the frames and how much detail there is per frame.

[hzeller]

Thanks, crash is fixed and I could test the animation now. TL;DR: animation works.

The animations now seem to work with iterm2 protocol. Even if I output things at the highest rate by not having any frame delay, wezterm seems to smoothly display things; here the parameter to send things as fast as timg can encode them:

  timg some-video.mov --pixelation=iterm --verbose --debug-no-frame-delay

When not emitting animations, but many images that make the terminal scroll, some update 'choppiness' is noticeable, as if the screen is filled and only updated every 1/10th of a second or so. Don't know if this has to do with the latest change or if it was always like that or can be configured independently.

You can try that if you have a bunch of images and want to show them

  timg *.jpg -pi --grid=3x1 --verbose

-pi short for --pixelation=iterm

On the Kitty terminal with their protocol, on the same size terminal, the scrolling feels a lot smoother (but in terms of absolute time both terminals are close; on my machine 8.7 sec wezterm, 6.2 sec Kitty for 482 pictures)

  timg *.jpg -pk --grid=3x1 --verbose

Anyway, I think w.r.t. to the original complaint about 'strong flickering', this issue can be closed.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.