support iTerm2 graphics protocol

[dankamongmen]

we have two thirds of the graphics triforce now, with sixel and kitty's protocols. it would be a fine thing to add support for iTerm2, though i have no way to develop or test with said terminal at this time.

[kovidgoyal]

Use WezTerm it implements it too.

[dankamongmen]

So it appears that this works on a file-based level, using either a URI(!) or base64-encoded file data. That's going to be a tremendous problem for any kind of scrubbing, egads.

[dankamongmen]

Well, not if it works with an uncompressed format like BMP; we could just throw a header around our RGBA data and be done with it (or uncompressed PNG).

[dankamongmen]

also, does that mean this just doesn't work over ssh? tres lame.

[dankamongmen]

well now that we support OS X, this becomes rather more important.

[dankamongmen]

i flagged down @gnachman over in #1821, and he graciously offered to help with any questions about the protocol. please reply only at your convenience, my friend.

so to bring any readers up to speed, notcurses integrates bitmaps into its multiplanar model with a calculus of "wipes" (removal of a cell's worth of data from an active graphic) and "restores" (restoration of a missing cell). these can happen out of order -- there is not generally nesting of wipes/restores. this is necessary to draw text in cells containing graphics without destroying the entire graphic (though it often requires redrawing the entire graphic--only kitty 0.20.0+ can do partial updates).

i just got Notcurses building on OS X tonight; i've never personally used OS X nor any Apple products. but now that we're there, i want first-tier support for your powerful iTerm2. unfortunately, i have no access to it. so, questions for you (@wez, i'd be very interested in any ways that WezTerm divergences from @gnachman's reference implementation):

(a) does your image protocol intend to honor transparency? it does not seem specified in the document. what does iTerm2 currently do with transparent pixels, when placed atop (a) glyphs, (b) other graphics, and (c) a combination of the two, if a combination is possible?

(b) is there any way to destroy the graphic aside from writing glyphs atop its entirety? if so, do these methods destroy the glyphs that are in the same area?

(c) it appears that glyphs are drawn at the current cursor position. do they result in screen scrolling? where does the cursor emerge? is this true if my text direction was other than right-to-left?

(d) does emitting a graphic reset or otherwise change any other text properties?

(e) i see the default filename is "Unnamed file". can i emit many graphics, all without filenames, without disturbing each other (so long as they don't intersect)?

thanks so much!

[dankamongmen]

there are two alternatives to wipes and restores: animation ala kitty (where i can update a subregion of an existing graphic), and "mosaics", where i draw the graphic using cell-sized chunks, resulting in many graphics. if iTerm2 is amenable to either of these solutions, that would be interesting.

[wez]

i'd be very interested in any ways that WezTerm divergences from gnachman's reference implementation

The protocol allows for general file transfer; wezterm doesn't implement that, only the image display portion. The intent is that wezterm is faithful to iTerm2's implementation, but I wouldn't be surprised if there were some accidental differences. If you find those, I'd love to fix them in order to be conforming!

[dankamongmen]

If you find those, I'd love to fix them in order to be conforming!

rest assured that bugs will continue being filed until notcurses can have its orca drive her segway across all the terminals of this earth

https://getyarn.io/yarn-clip/28c6e15f-4679-4b70-9ac3-57f5ff20fb56

is there any situation for which Red Dawn doesn't have something applicable? ahhhh

[dankamongmen]

there doesn't appear to be any way to check for support of this protocol that i can find, unfortunately. i guess we'll just key off TERMINAL_ITERM2, as detected via XTVERSION. wrt wezterm...i guess let's drive this with wezterm for now, just so we can get it tested. maybe then we stick with it. i like the absence of quantization for sure.

[dankamongmen]

so i've got basic image display just about working, but wipes and restores are going to be a tremendous pain in the ass due to the structure of PNGs. essentially, you've got the following hierarchy:

• the PNG is base64-encoded for iTerm's protocol
• the PNG is formed of (among other things) one or more IDAT chunks
• the data within these chunks is deflated
• the data is split into scanlines

so if we wanted to edit this in place, we'd have to determine the relevant chunks, inflate them, patch the data, deflate, fix up the chunk length and crc, and reencode.

right now we're making maximum-sized chunks. we could simplify things by making a chunk per row, at the cost of some overhead. but that doesn't address the deflate/inflate issue -- we'd likewise have to finalize the deflation at each scanline, (probably) degrading overall compression efficiency.

yuck.

this might call for mosaics. i don't want to fuck with all this shit. with a mosaic approach, this becomes simple, if a bit more heavyweight.

let's get the simple display tested and merged, and punt the rest for later.