dankamongmen / notcurses

blingful character graphics/TUI library. definitely not curses.
https://nick-black.com/dankwiki/index.php/Notcurses
Other
3.49k stars 112 forks source link

make better use of kitty's graphic underlay #1444

Open dankamongmen opened 3 years ago

dankamongmen commented 3 years ago

See #1388. We need the whole kitty_null() system to get equivalent behavior from sixel and kitty protocols, but we can do better with kitty, at least when the image has no transparency: we can put it "underneath" all text (with z=-1). we can then logically implement #1388 by clearing out any text logically underneath the image, and dispense with kitty_null() iff the text logically above a graphic has a transparent background. it will look awesome when it plays out, and it doesn't break the shared abstraction.

it's not high priority, as it only benefits kitty, but it'll be phat.

dankamongmen commented 3 years ago

if we're going to do this, figure out any necessary API changes by 3.0, and ideally earlier than that.

dankamongmen commented 3 years ago

so the acceptance criteria for this would be, i think: in intro, when this is working, we ought be able to see the orca under the HUD but over the deltas.

dankamongmen commented 3 years ago

so the acceptance criteria for this would be, i think: in intro, when this is working, we ought be able to see the orca under the HUD but over the deltas.

(because right now you can't see it under the HUD, which ought be translucent)

dankamongmen commented 3 years ago

so we have the following cases:

on the plus side, i see no need for API changes -- the existing multiplanar compositing system admits all these situations.

so how would this work, assuming we broke up the image? we would have to be able to change the z level of a tile dynamically. imagine that we have a glyph:

G

and then we draw a partially transparent bitmap cell atop it

G B

at this point, the bitmap must be at z=0. we then create a new pile that draw a glyph above the bitmap cell with a transparent background (this clears G):

B 2

z here must be negative. the top plane is then moved away:

G B

z must go back to 0. so we can specify z in the placement command, which is good, that keeps things simple (no mucking with an existing encoded sprixel). but we would need to communicate the desired z to the placement command from rendering, which means we'd need add it to the sprixel_commit() argument iirc.

but that's not hard. the complex thing would be moving to mosaics. moving to mosaics requires performance from the emulator, and also pretty much requires a partitioning data strucutre (to avoid an extra big O(N) in the rendering pass).

dankamongmen commented 3 years ago

so i think what this calls for is going ahead and moving to full mosaic mode for kitty graphics. let's see how kitty and wezterm handle it. this would be a 2-series of sprixcells all in a single plane.

let's not do this for sixel just yet. sixel implementations are weaker overall than kitty and wezterm, there's more overhead with sixel due to cells not necessarily being multiples of 6, etc (though it would help quantization). let's just try it out with kitty graphics.

dankamongmen commented 2 years ago

so obviously full mosaics are the end target, but for 3.0.0, how far can we get with opaque images? i think it would be a fair amount of logic, especially since in multiframe media you could have an opaque become partially transparent in the next frame, or the other way around. you'd need new states in the sprixcell_e enum, and substantial new code to determine whether a wipe was necessary or not.

so i'd rather just pursue mosaics in notcurses iv.