problems with demo intro in iTerm2

[michaelsbradleyjr]

It's been awhile since I ran the demo from the start in iTerm2. Below is a recording of what I'm seeing today (build of current master). In Kitty it looks correct.

https://user-images.githubusercontent.com/194260/126908478-2fc6b479-a5da-4050-8723-72e147f4c50d.mp4

[dankamongmen]

hrmmmm was it properly doing transparency before? problems include:

• not clearing up old draw (probably need iterm_remove() or iterm_scrub()
• not being transparent. does iterm do transparency? @gnachman never made this clear? or are we generating bad PNGs? wezterm now has transparency, and implements iterm protocol, so i ought be able to find out there
• what's all that garbage on the screen at the end?

i'm thinking none of this is regression -- are you sure it was working better in iterm2 before?

[michaelsbradleyjr]

are you sure it was working better in iterm2 before?

I went back quite a few commits to when support for macOS landed in master, tried some commits between that one and 757fb5811f72129ea21a663125ea552c456dfd8a. Orca-on-segway doesn't appear until after the latter and displays the same as in my screen recording.

So I probably confused myself with memories of it working correctly in Kitty at that time (still does).

[dankamongmen]

cool, thanks for looking into it, and i'd always rather have a bug report that turns out false than not get one!

[dankamongmen]

hrmm, based on the screencapture here https://iterm2.com/img/screenshots/v3-screen-shots/iterm2-inline-images-demo.gif, it seems iTerm2 should be able to take advantage of transparency? maybe? at least with GIFs? some experimentation is necessary.

[gnachman]

Transparency doesn't support overlapping images, unfortunately. It lets the background color show through.

[dankamongmen]

Transparency doesn't support overlapping images, unfortunately. It lets the background color show through.

there's no overlapping image at work here -- the orca is atop a bunch of glyphs (except where there are glyphs atop the orca); the orca is the only bitmap.

[gnachman]

A cell can't contain both a character and an image. Transparency really just shows the background color through.

[dankamongmen]

A cell can't contain both a character and an image. Transparency really just shows the background color through.

ACK. ok, in that case, Notcurses graphics are never going to work properly on iTerm. combined with the weird effect of graphic display involving a file-save prompt, i will likely just remove support for the iTerm protocol from Notcurses.

@gnachman , i humbly ask you to revisit this decision; my arguments are best stated in the bug i filed on WezTerm regarding this same issue: https://github.com/wez/wezterm/issues/942. basically, there's a lot a client can do if graphics and glyphs can share a cell, whereas the alternative presents comparatively little room for interesting work. it is of course your project, and i respect your decisions.

if you've never run notcurses-demo in a terminal like WezTerm, XTerm, or Kitty and seen what it can do, you might want to -- or check out the Notcurses III video here: https://www.youtube.com/watch?v=dcjkezf1ARY.

in the meantime, i'm going to go ahead and disable iTerm2 protocol support, and call this bug closed.

[gnachman]

This is a reasonable request. I am running out of bits to store info in the per-cell struct and I need to add support for out-of-band data to handle cases like this. It's a big project which I hope to get to this year.

[dankamongmen]

This is a reasonable request. I am running out of bits to store info in the per-cell struct and I need to add support for out-of-band data to handle cases like this. It's a big project which I hope to get to this year.

this is fantastic news, and frankly unexpected. you've made me a very happy orca today, @gnachman !

[dankamongmen]

This is a reasonable request. I am running out of bits to store info in the per-cell struct and I need to add support for out-of-band data to handle cases like this. It's a big project which I hope to get to this year.

just wondering if there is a bug on this? searches were timing out on the iterm2 gitlab this morning.

[gnachman]

I need more info on the capabilities you want. I opened an issue here: https://gitlab.com/gnachman/iterm2/-/issues/9989

Would supporting the Kitty image control sequences give you what you need?

Do you need images to overlap images? If so, how many layers deep?

Do you need images to overlap text?

[gnachman]

Oh, and the oob storage is done so now it's a simple matter of programming :)

[dankamongmen]

HOW EXCITING. i'll answer there as well, but:

(a) I do think the Kitty protocol to be superior to known alternatives, but even your current protocol would work for me, so long as it provided transparency atop text where the image was transparent (b) images will sometimes overlap images, but there's no way to indicate relative "height" in either the Sixel or iTerm2 protocols, so i've currently got that implemented at 'best effort' status. if you do support kitty, it does support arbitrary heights (for graphics only), and it would be appreciated if that was honored. (c) yes, images atop text is the primary thing i need (see (a)).

[gnachman]

Images overlapping images is going to be hard to do, so I'd rather skip that unless you have a strong need for it (or unless the Kitty protocol requires it, which I haven't investigated yet)

[dankamongmen]

Images overlapping images is going to be hard to do, so I'd rather skip that unless you have a strong need for it (or unless the Kitty protocol requires it, which I haven't investigated yet)

kitty does, but as stated over on your bugtracker (https://gitlab.com/gnachman/iterm2/-/issues/9989), it's not of tremendous importance to me. it's a definite nice-to-have, but if worst comes to worst, i can compose images myself internally.

[ghost]

@gnachman I recently implement images-overlapping-text, including images-overlapping-images. If your internal model is similar to mine (image fragments (or references to slices) passed around in text cells), you might find these notes relevant (the bulleted list): https://jexer.sourceforge.io/evolution.html#year2021

The big performance boost for me was identifying what would be a fully-transparent image over a text cell, and then just disregarding the image. Doing just that alone -- even without glyph-under-image -- can be pretty usable:

But with the glyph-under-image, and then image-over-image, it gets really nice.

[dankamongmen]

If your internal model is similar to mine (image fragments (or references to slices) passed around in text cells)

@klamonte as i understand it, you use horizontal slices of images rather than cell slices. what's your reasoning here? i'm guessing less overhead due to per-cell palette transmission is your primary goal? or was it compensating for terminal algorithms running in time proportional to the number of images? i intend to move to cell-sized units soon, as they greatly simplify the logic of the z-axis.

[ghost]

@dankamongmen

The model is like this:

• A Cell can have text AND image data. The text is a single codepoint (boo! This is my biggest gap in the notcurses demo). The image is a "cell-sized" RGBA bitmap. When an image is set on a cell, the hash code for that image is computed, and the flag indicating whether it has transparent pixels is reset to "unknown".
• The internal cell size in pixels for single-head applications is the same as that provided by the terminal. For multi-head applications, it is a virtual size that is the smallest size in each height/width dimension for all the attached heads. The result of this is that images can get bigger when scaled to an output screen, but never smaller -- hopefully preventing missing information.
• The logical screen is built up on every frame by having every widget/window draw in reverse-Z-order (the coordinate space is right-handed, so positive Z is away from the user). Any cell can cover any other cell during this. During this phase the Cell might have an image transparency check: if it does then the result of that is cached. This puts all the penalty on the first frame: on subsequent frames each widget will "know" which cells are transparent.
• Currently there are only two paths for a partially-transparent image to cover another image, in the "tackboard" which can place arbitrary bitmaps anywhere: https://gitlab.com/klamonte/jexer/-/blob/master/src/jexer/tackboard/Tackboard.java#L266 ; and in the terminal widget: https://gitlab.com/klamonte/jexer/-/blob/master/src/jexer/tterminal/ECMA48.java#L8445 . At these points it blindly blits one image over the other - which conveniently handles alpha blending too. All other paths to generate image data will either set a Cell to text, or an image, but not an image-over-image -- that's just an artifact of how I got here though and not a specific design choice. If I have another widget/object that can put image-over-image, then I will need to handle image-over-image at that spot too.
• Once the logical screen is fully assembled, it goes to the output heads which can be a mix of GUI/Swing or Xterm screens.
• For a GUI/Swing screen, it has to build the whole frame (it's triple-buffered) and then push to the actual OS.
• For an Xterm screen, ideally only the changed cells (text or image) will be output. I output all images first, and then all text - this seems to produce the least flicker on real terminals. (But now I also do Synchronized Output when I detect support for that.)
• For cells containing images, consecutive horizontal cells with images are concatenated into one image. For multihead, that image is then rescaled to match the cell aspect ratio of the final terminal.
• The (maybe-rescaled) image is then converted to sixel/Jexer/iTerm2, and the final string (minus cursor position and DCS/ST start/end) in put into a LRU cache. If the same string of cells comes in, the cached string is output instead with its own cursor postion and DCS/ST. (Checking cell equality is also fast because I have cached the image hash code.)
• The result is that no image is ever taller than a single text cell, but can extend across the entire screen (or 1000 max pixels due to xterm). It can also be as small as a single cell.
• Any rows that contained image data that have any different cells on this frame will have all of the images on that row re-drawn, in case some of the text I drew earlier obscured an image from some previous frame. It's really unknowable what the terminal will do to partially-overwritten images, so I take that choice away from them and blow every potentially-different image away.

I settled on the horizontal strips option because I could not easily figure out a damage map that would produce an optimal number of rectangular images to cover the changed cells, while also ensuring that images from any previous frame that were covered by text in this frame are also fully included in said damage map. With the strips, I still have a lot fewer palette copies to push out than per-cell so gain most of the throughput benefit over per-cell, but more than that I can easily reason about what I will actually redraw.

[ghost]

Also: I hope to soon have the energy to get serious about building decent palettes for sixel -- thanks to someone else doing a bang-up job on that recently ;-) -- and I'm genuinely curious how the registers:colors ratio works out for strips vs large images, and if that would lead to visible horizontal banding artifacts if I use different palettes (even if more optimal).