figure out pixel integration with the z-axis

[dankamongmen]

I first bring this up in #1380, but it really demands its own issue. How are we going to integrate the realities of Sixel with our z-axis/transparency model? Sprixels are written out atomically, covering whatever pixels they don't specify (i.e. not necessarily blowing away entire cells). Cells written atop a sprixel are blown away entirely.

I don't really see any way to do our entire z-axis model. Sprixels atop cells work just fine, but we can't write glyphs with transparent backgrounds atop sixels and retain them, at least so far as I can see. So the simple interpretations is simply: sprixels aren't integrated into the model. If another cell is atop sixels, the sixels don't get factored in. If sixels are atop other cells, the cells below aren't factored in.

But that's kinda lame, since we do have transparency within a sprixel. BUT doing this properly is complex: we'd have to emit all the cells underneath the sprixel, and only then draw the sprixel atop them. BUT you can't just emit the sprixel after writing everything but sprixels, because then you blow away things that were above the sixels. You almost have to layer them: cells below sprixels, sprixels, cells above sprixels. And if you have sprixels at multiple levels of the z-axis, this quickly gets to be a pretty big pain in the ass.

For 2.2.3, I think it's sufficient to get something simple up, but this problem demands hard thought.

[dankamongmen]

We see this problem even already in the keller demo (see #1381). When we print "pixel", we only see "pixe". I suspect this due to it printing over "braille". Note that the fifth character in both is "l", and thus this character would not be marked as damaged. Thus it isn't printed over the sixel. But we can't just mark everything as damaged, or else we reprint it all, and the sprixel is destroyed. =[

[dankamongmen]

...hrmm, maybe not. i change the foreground color each iteration, which ought cause damage, and yet we still see the same result :(.

[dankamongmen]

there's the complicating factor that the procedures and indeed capabilities are different between sixel and the kitty protocol. they're entirely different systems.

• the sixel method is temporal -- newer material trumps older material. print a sixel atop text (or another sixel), and you get the sixel, modulo any omitted pixels. print text atop a sixel, and get the text (other cells are unaffected; the printed cells are annihilated).
• the kitty method is positional -- images can be placed on a z-axis, but all text is on z=0. a negative z means the image is drawn underneath all text. but there seems no way to have some text underneath, and some text above, a kitty image. so even if you drew with RGBA so you could leave some pixels of text elements showing from underneath, you then couldn't draw text above an image :(. with that said, different images can be put on different zs, so you could have one image above text and another below text.
  • kitty lets you do partial placements and multiple copies of an image, so you could technically synthesize the parts of an image around the text you wanted to draw "atop" it. this would be...a tremendous pain in the ass, but it's doable. and this would match the sixel behavior, where printing atop the image results in the cells being wholly annihilated. so this solution would involve a) always drawing images at z>=0 b) drawing underlying text first c) drawing images with RGBA and d) "cutting away" from images to draw text. alternatively, maybe it is possible to have the "cut away" region be a redrawn version of the image, with a transparent section? oohhhhhhhhhh @kovidgoyal have you any better suggestions?

[kovidgoyal]

Not sure I understand the problem. Why do kitty and sixel semantics have to be identical? I dont think that's even possible, since sixel doesnt event support full resolution images. So why does it matter that they also differ when printing text over images? Simply have that behavior be undefined and clients that rely on it are broken. Or if notcurses is keeping track of image and cursor positions, have the library throw an error if it detects text will overlap an image. Maybe have a way for clients to override that error and force text over image, with the caveat that behavior will be different based on the imaging backend.

[dankamongmen]

they don't have to be identical, of course, but i'm trying to offer an abstraction layer atop the two. if someone's blitting to a plane with NCBLIT_PIXEL, and putting that between two text planes, i want the behavior to be the same regardless of the terminal (so long as pixel graphics are supported). even in the absence of sixel, i'd want to be able to have text both "above" and "below" the image.

with that said, only with the kitty protocol can i sanely stack image graphics, as far as i'm aware, so that's a definite plus -- i think that'll just break under most sixel implementations. but i'm going to make it available nonetheless.

btw here's some nice text above an image using kitty

https://youtu.be/po9vixBt08E?t=31

[kovidgoyal]

So in kitty you can have text above and below different images. I think what you are asking for is to give text z-indices too. I dont think that's a bridge I am ready to cross in kitty. You could simulate its effects by chopping up the image per cell and setting different z-indices on each cell image.

[kovidgoyal]

And I should add w.r.t. chopping up images into cell sized pieces, this is not a scenario the data structures that track images in kitty are optimized for, so I dont know what the performance would be like.

[dankamongmen]

i think i've got a workable scheme here:

Fundamental facts

• Printing atop pixels with sixel shows the new text, at the cell level.
• Printing pixels atop text with sixel covers it, except where transparent.
• Reprinting sixel graphics restores them.
• Printing kitty graphics wipes out text, new and old, if z=1, except where transparent.
• Printing kitty graphics goes under text, new and old, if z=-1.
• Kitty graphics can be deleted, restoring what was under them.
• Kitty graphics, once loaded, can be quickly replayed.
• Sixel graphics cannot be deleted.
• Reprinting kitty graphics in the same location is pointless.

Classes

• Text can be under pixels when they're first drawn. This is only important where there are transparent pixels.
• Text under pixels can be updated. This is only important where there are transparent pixels, except in sixel, when they must not be drawn.

Actions

• When a pixel graphic is removed, we must:
  • kitty: delete the instance
  • sixel: invalidate all cells it covers
• When a pixel graphic is moved, we must:
  • kitty: delete the instance and redraw it
  • sixel: invalidate all cells covered only by the old position
• When a cell underneath a transparent pixel graphic is updated, we must:
  • kitty: invalidate it
  • sixel: invalidate it and redraw affected pixels
  • we will likely do this regardless of the transparency, at least for now
• When a cell above a pixel graphic is updated, we must:
  • kitty: delete, make the section transparent, and redraw
  • UNLESS the section is already transparent, in which just invalidate
  • sixel: invalidate it

we'll want to take advantage of kitty's ability to load and redraw an image; we have no such luck with sixel, and must resent the image in its case, lame.

[dankamongmen]

To do this sanely, I think we'll take the following steps:

• pixels can only be rendered to a new plane (ncvisual_render(NCBLIT_PIXEL) requires n be NULL)
• a pixel plane is immutable -- it can be moved or deleted, but not otherwise modified
• we'll mark such planes, thus exposing them to the rendering engine
• the plane's dimensions themselves are thus guaranteed to match the pixel graphic's, for easy invalidation of 2D areas

[dnkl]

we'll want to take advantage of kitty's ability to load and redraw an image; we have no such luck with sixel, and must resent the image in its case, lame.

You don' have to redraw the whole image, only the part(s) that has been erased/overwritten. I believe this is one of the reasons Jexer splits up images into stripes mapped to cell rows.

FWIW, at least XTerm (and foot), will not include empty pixel rows in the last sixel row in the final image, meaning you can emit images with a height that isn't a multiple of 6. But this is likely an area where different terminals behave differently.

[dankamongmen]

FWIW, at least XTerm (and foot), will not include empty pixel rows in the last sixel row in the final image, meaning you can emit images with a height that isn't a multiple of 6. But this is likely an area where different terminals behave differently.

hrmmm, this seems to contradict my own xterm experience. let me go verify....

[dankamongmen]

alright, i think the thing to do is to keep a two-bit-per-cell map of each sprixel in the sprixel cache. for each cell, we want to know if it's (0) opaque, (1) transparent or semi-transparent or (2) annihilated. that allows us to quickly perform mop-up invalidations and redraws as necessary, and at a per-cell granularity. Also, I'm wondering whether we can't make use of kitty's full 8 bits of alpha to effect an image above some text and (translucently) below others....i think it will work. Finally making progress here!

[dankamongmen]

OK, after the work on #1401, we're getting pretty close.

• [x] with sixel, we still need to redraw when we redraw a cell behind a transparent part of the sprixel. For this, we need sprixel invalidation from the raster core; the call to rasterize_sprixels() will then naturally handle things. Bonus points for using a transparency map and thus only redrawing necessary parts of the sprixel.
• [ ] ~with sixel, when we rasterize a pixel graphic underneath a text element which was damaged during the same rendering cycle, we need update it after the pixel raster -- we can do it at both times, but it must at least happen afterwards.~ we're instead going to perform surgery on the sixels inline, as we do kitty
• [x] with kitty, we need a way to print atop an image at z<0, which looks like it will be cutting out an α=0 hole where we want the text. we never need to redraw a kitty image, even if we update the cell underneath a transparent area.
• [ ] we need restore content after it's no longer occluded, see #1440

it would also be very good to use kitty's presentation commands to move around an image in response to plane moves, but that's more for #1395.

[dankamongmen]

so the next step is to make the transparency/annihilation map for each sprixel, as mentioned above. let's go into some more detail. for each cell in the ncplane, on initialization we set up a transparency map, marking a cell 0 if no pixels are transparent, and 1 if at least one pixel is. we'd need build this map up in refine_color_table() for sixel and base64_rgb3() for kitty. for libsixel, we'd just need to do an additional pre- or postpass, yuck.

we still need to determine the intersection of the rendering area and these transparency maps, which sounds messy. ideally we'd drive this from the sprixel pass, so that we only ever do work when sprixels are in play. to do that, we'd need stop setting damaged to 0 inside rasterization, which we currently do to support sprixel-based damage. so the sprixel pass would have to go through and touch everyone, at which point it becomes better to do from the rasterization pass itself.

[dankamongmen]

...YOU KNOW...

kitty's transparent text atop graphics looks awesome, no doubt about it. much cooler than the sixel equivalent (z==-1):

and it does sit "atop" the background color...and it vastly simplifies things...the only problem is that it teaches you that you can draw text all over a graphic and it'll work, which it won't for sixel...but that's fine. we can just say "text with a transparent background might or might not work; it might blow away chunks of the image it's drawn on". so they avoid it except where it's necessary...and then maybe other terminals catch up in the future, hrmm.

in this case we still need to redraw under the graphic (followed by a redraw of the graphic where affected) for changes beneath the graphic, but we don't have to do anything for text atop it. we don't need to track annihilation, we don't have to do null-alpha cuts on kitty graphics, that's all much simpler. and in kitty and kitty-like protocols, things can look much cooler. hrmmmmm. you would have to wipe out all text underneath opaque parts of a kitty image manually, bleh. hrmm, also, with kitty, if you wanted a background up above the image, that apparently requires an alpha cut, since it's sticking the background color behind everything else, see below:

[kovidgoyal]

You can have background above image by specifying a suitably negative z-index for the image. Quoting from the spec:

Finally, you can specify the image z-index, i.e. the vertical stacking order. Images placed in the same location with different z-index values will be blended if they are semi-transparent. You can specify z-index values using the z key. Negative z-index values mean that the images will be drawn under the text. This allows rendering of text on top of images. Negative z-index values below INT32_MIN/2 (-1,073,741,824) will be drawn under cells with non-default background colors.

[dankamongmen]

Finally, you can specify the image z-index, i.e. the vertical stacking order. Images placed in the same location with different z-index values will be blended if they are semi-transparent. You can specify z-index values using the z key. Negative z-index values mean that the images will be drawn under the text. This allows rendering of text on top of images. Negative z-index values below INT32_MIN/2 (-1,073,741,824) will be drawn under cells with non-default background colors.

you're absolutely right, of course; thanks for the heads-up! i'd read this, but dismissed it as not relevant to my needs, but with this possible new approach, it suddenly becomes useful.

[dankamongmen]

Finally, you can specify the image z-index, i.e. the vertical stacking order. Images placed in the same location with different z-index values will be blended if they are semi-transparent. You can specify z-index values using the z key. Negative z-index values mean that the images will be drawn under the text. This allows rendering of text on top of images. Negative z-index values below INT32_MIN/2 (-1,073,741,824) will be drawn under cells with non-default background colors.

of course, if i do this, i can't have text show behind (only) transparent regions of the same graphic. =[ i.e. i can't have a single graphic whose transparent regions show text (while being opaque in the pixels they cover, in the same cell) while also allowing text to draw foreground+background on other regions of the graphic. that's the fundamental issue i have with the kitty protocol, which is otherwise a tremendous improvement over sixel imho.

[kovidgoyal]

On Thu, Mar 18, 2021 at 04:56:06AM -0700, Nick Black wrote:

Finally, you can specify the image z-index, i.e. the vertical stacking order. Images placed in the same location with different z-index values will be blended if they are semi-transparent. You can specify z-index values using the z key. Negative z-index values mean that the images will be drawn under the text. This allows rendering of text on top of images. Negative z-index values below INT32_MIN/2 (-1,073,741,824) will be drawn under cells with non-default background colors.

of course, if i do this, i can't have text show behind (only) transparent regions of the same graphic. =[ i.e. i can't have a single graphic whose transparent regions show text (while being opaque in the pixels they cover, in the same cell) while also allowing text to draw foreground+background on other regions of the graphic. that's the fundamental issue i have with the kitty protocol, which is otherwise a tremendous improvement over sixel imho.

As I said before, that would require giving text also z-indices. That is not something I am willing to do. Without it, you have the following options to achieve it:

1) Cut up the image into sub-images 2) Render the text as an image yourself and blend as desired

[dankamongmen]

As I said before, that would require giving text also z-indices. That is not something I am willing to do. Without it, you have the following options to achieve it: 1) Cut up the image into sub-images 2) Render the text as an image yourself and blend as desired

yep, i get it =] and cutting up the image (or setting a region of it to alpha=0) is exactly what i'm planning.

[dankamongmen]

holy jesus, i just had a realization: if we cut the picture down to component cells, you could 1 always have a perfect set of color registers in sixel, and 2 deal with pixels like you do any other cell-oriented thing. we'd just need to make sure terminals can deal with thousands of images at the same time. there'd also be significant overhead due to loss of RLE, fixed sprixel setup, and repeated color registers. but yeah, do that, and suddenly we're rid of all this special-purpose code. need to do the experiment first, though.

[kovidgoyal]

See my comment from last week: https://github.com/dankamongmen/notcurses/issues/1388#issuecomment-794966381

[dankamongmen]

yep. well, look forward to some experimental data from me =].

[dankamongmen]

For reasons detailed in #1422, we will not be taking the cut-up method to the limit and breaking bitmap graphics down into cell-sized components. Instead, we're doing in-place editing of glyphs, because I'm a madman. I've just implemented the kitty half of this in #1427. Sixel incoming.

[dankamongmen]

Success, eureka, first light! We are now printing atop a z=1 kitty graphic with transparencies, showing the background where we ought, and the text where we ought, and the graphic where we ought. Needs more work, and needs Sixel completion, but the basic idea is proven out.

https://www.youtube.com/watch?v=6jJkdRaa04g

[dankamongmen]

i mean it for real looks like shit right now, and i clearly have a bug in my kitty scrubber when you look at those sawtooth patterns at the bottom, and it's slow as fuck (this will change), but the fundamental principle works. this kitty graphic could not normally be drawn on; we're scrubbing the sections that we print on top of.

we know kitty can display the thing quickly enough, since we're already updating it for the infection spread (this is the yield demo). it's my kitty_wipe_cell() that's slow, but that's expected, and we're going to solve it with the transparency-annihilation cache. easy peasy.

we've got some hats now, motherfuckers!

seriously it will run waaaay better than this very soon

https://user-images.githubusercontent.com/143473/111880585-c2176900-8982-11eb-9243-ac5bb1ec6cb0.mov

[dankamongmen]

Got it working in Sixel as well, now =] =D

[dankamongmen]

already improving

https://user-images.githubusercontent.com/143473/111882797-e6c50e00-898d-11eb-9847-39410cf92e8d.mov

[dankamongmen]

i need to take another look at the animation protocol and see if we can use it to do a minimum transfer+redraw when we knock one cell out like this.

[dankamongmen]

https://user-images.githubusercontent.com/143473/111883290-bdf24800-8990-11eb-9b73-b55d531de6e5.mov

[dankamongmen]

w00t!

[dankamongmen]

the yield demo works just fine (at least in kitty) if we hide the FPS graph -- that flickering goes away. we're still redrawing the graphic as new countries turn red, so it must be somehow related to cell wipes and the printing atop the graphic.

i wouldn't be surprised if a large part is due to slowness in the wiping. when we put the cache in, things got massively better.

xterm looks kinda ehhh either way, sadly.

[dankamongmen]

https://www.youtube.com/watch?v=Uh8jV-evm1E looking just about perfect with kitty now

[dankamongmen]

We're now building up the tacache from the constructors, and passing it into sprixel_create(). We're not yet setting up the Ts, so we should add that. Currently it's initialized to 0s.

[dankamongmen]

i've implemented the invalidation for when a cell underneath a bitmap is updated, fixing pixel for Sixel terminals. we now really need to get sixel_wipe_cell() working and hooked up -- until we do, yield looks like shit (the FPS graph keeps getting hidden) and keller is off (the last drawn bitmap stays up past its time). sprite_sixel_annihilate() doesn't seem to be doing its job (and also leads to flicker; see #1454 ).

[dankamongmen]

Still need a working sprite_sixel_wipe_cell(), which eludes us. We can construct a proper chopped- and screwed Sixel afresh when bringing forward the surgery map (see #1457), but we can't seem to properly erase one.

We need do some experimentation (at the shell) to lock down exactly how Sixels work when you print one with a missing portion over another.

ncplayer almost works, we just don't see glyphs rendered prior to the sixel. yield likewise almost works now, but we don't block across the entirety of the fps graph. i think maybe you have to blit a glyph when you cut out a section, not just replay the modified sixel? lock this down before continuing to flail around. come up with a rigorous experiement.

[dankamongmen]

OK, i know the reason why sprite_sixel_wipe_cell() doesn't work, after so many wasted hours: my experimentation (and consulting with @dnkl ) demonstrate that you can't scrub a Sixel by emitting empty cells. Well, shit. That's not a huge problem, though, because we can always just overprint in Sixel. The only big problem here is that we must indeed reprint any affected glyph every time we redraw the sixel. Right?

[dankamongmen]

....i think i might have got it. Everything appears to be working perfectly in both sixel and kitty. Doing some final testing...