need restore bitmaps when no longer occluded

dankamongmen commented 3 years ago

Sprixel sprixcells can be ANNIHILATED, but what happens when the annihilating cell is gone? We need redraw the sprixcell, and restore it to its former state (of OPAQUE, MIXED, or TRANSPARENT). This means we need

detect this transition,
keep around the materials necessary for reconstruction, and
be able to build the sprixcell back up, ideally in-place

To detect the transition, we'll add code to paint_sprixel() that checks for an ANNIHILATED sprixcell without a glyph having already been solved. In such a case, the sprixcell cannot be ANNIHILATED, and ought be restored.

To keep the materials around, we will add a void* to each cell of the TAM. Upon moving to ANNIHILATED, the necessary matrix is copied out into the TAM. For Kitty, this will be a matrix of alpha values (since we drop them all to 0 for a wipe). For Sixel, this will be a matrix of RGBA (since they're completely elided). In both cases, we'll need get this information while doing a sprixcell wipe. This has the side effect of being able to tell when a cell has been marked ANNIHILATED but not actually wiped (it will have a NULL matrix).

Rebuilding the bitmap is trivial for kitty -- just copy the alpha values back and free the matrix. For Sixel, it'll be a bit more complicated, and we'll likely want to repair as many sprixcells in a single go as we can (since we can't O(1) index into sixel). For 2.3.0, it might be good enough to do each by itself, but i think we'll want to batch (the worst case doesn't allow us to batch, so be prepared for it anyway).

For the love of god, do some unit tests.

things we need:

~[ ] write_kitty_data() needs to rebuild auxiliary vectors~ (#1605)
[x] kitty_rebuild() need be written to reinsert pixels from auxiliary vectors
[x] kitty_wipe() needs to build auxiliary vectors
~[ ] sixel_blit() needs to rebuild auxiliary vectors~ (#1605)
[x] sixel_wipe() needs to build auxiliary vectors
[x] sixel_rebuild() needs be written to reinsert pixels from auxiliary vectors
[x] sixel needs a bit vector for transparency in auxvec (currently we rip off one color)
[x] implement the TRANSPARENT -> ANNIHILATION_TRANS optimization described i #1587

dankamongmen commented 3 years ago

we need implement kitty_rebuild() and sixel_rebuild(), but they're plugged in and ready to go.

dankamongmen commented 3 years ago

one other thing: a byte is insufficient for the Sixel-form auxiliary vector. we need at least 8 bits to cover a 256 entry palette, but we also need a single bit to indicate presence or absence of the pixel. so it gets 16-bit auxvec entries, which is nice anyway because it means we can deal with larger palettes (up to 32K) without modifications.

dankamongmen commented 3 years ago

this is interesting:

Sprixel 1 (0x5555556358d0) 26x30 (515x300) @43/44 state: 3
444444444333334444444444444444
444444443311134444444444444444
444444433111133444444444444444
444444431111113444444444444444
444433331111111344444444444444
443331111111111334444444444444
433111111111111133444444444444
311111111111111113344444444444
333333333111111111333333133334
444443333111111111111133334444
444444333311111111113334444444
444444433111111111113344444444
444444444331111111111334444444
444444444433331111111133444444
444444444443333311111113344444
444444444444333333111111334444
444444444444433333111111133444
444444444444433111111111113344
444444444444431111111111111334
444444444444311111111111111133
444444444444311111111111111113
555555555555555555555555555555
555555555555555555555555555555
555555555555555555555555555555
555555555555555555555555555555
555555555555555555555555555555
REBUILDING AT 21/0

that rebuild at 21/0 while still under the fps graph doesn't make much sense to me....wait, yeah it does, we're coming out from underneath the fps graph at this point, and this is precisely where reconstruction ought proceed. ok, good!

dankamongmen commented 3 years ago

bah, apparently we've been nulling out all of the RGBA pixel, not just the A, in kitty. this worked, but it won't anymore. so i need change up kitty_null() to retain RGB unmodified.

dankamongmen commented 3 years ago

one other thing: we also need build up a replacement auxvec for each ANNIHILATED cell when we load a new sprixel and have it recycle the preexisting TAM. ugh, so much work!

dankamongmen commented 3 years ago

first go at kitty rebuild is looking more or less correct!

dankamongmen commented 3 years ago

2021-04-26-012309_1463x1417_scrot

looks like we might have a bug in kitty_restore(), look at the pattern in the rebuilt sprixcells...

dankamongmen commented 3 years ago

got kitty done -- had a dumb bug in b64idx(). huzzah!

dankamongmen commented 3 years ago

Given that in #1452 we're already thinking about a quantization method that requires a final matching pass, maybe the easiest thing to do here is to just feed the thing back through Sixel generation? Editing it inline is going to be hellacious. We can forego palette quantization (we already have a palette) and just match all the pixels. This code can then be directly reused as the final pass in a new quantization scheme. Yeah, I'm doing that, and if it's too slow, we can speed it up later.

dankamongmen commented 3 years ago

Given that in #1452 we're already thinking about a quantization method that requires a final matching pass, maybe the easiest thing to do here is to just feed the thing back through Sixel generation? Editing it inline is going to be hellacious. We can forego palette quantization (we already have a palette) and just match all the pixels. This code can then be directly reused as the final pass in a new quantization scheme. Yeah, I'm doing that, and if it's too slow, we can speed it up later.

this could also eliminate the future CVE generator sixel_deepclean(), could it not? since we have to sideeye the TAM anyway.

dankamongmen commented 3 years ago

Given that in #1452 we're already thinking about a quantization method that requires a final matching pass, maybe the easiest thing to do here is to just feed the thing back through Sixel generation? Editing it inline is going to be hellacious. We can forego palette quantization (we already have a palette) and just match all the pixels. This code can then be directly reused as the final pass in a new quantization scheme. Yeah, I'm doing that, and if it's too slow, we can speed it up later.

this could also eliminate the future CVE generator sixel_deepclean(), could it not? since we have to sideeye the TAM anyway.

eh except we need to build auxvecs here. leave it for now.

dankamongmen commented 3 years ago

eh except we need to build auxvecs here. leave it for now.

but we're gonna have to update auxvecs when encoding a new Sixel under the same TAM. sooooo it would be best to go ahead and put that logic in sixel_blit() and junk deepclean_sixel(). except how will we handle the case where we're wiped in two different renders, in different places? if we always update on ANNIHILATED, the first one will set real values, but the second one will set all zeroes. we can't just say "OR in the previous value", because that will break updates for new frames.

fuck, this almost calls for yet another new state, PREANNIHILATION or something. no, we still have to update on ANNIHILATED for new frames, so that doesn't work. we want to update the auxvec:

when the cell is annihilated and
when a new frame is loaded

but not when doing an arbitrary annihilation scrub pass. ok, so we could add a flag to the whole process, set only for true sixel_blit() calls (new frame). we update the auxvec when this flag is set, or when the auxvec is not present, but not when the auxvec already exists and this flag is unset. that's gross, but it works....and then we eliminate deepclean_sixel().

dankamongmen commented 3 years ago

oooh if we do this, make sure we can always take advantage of the existing palette (we'll want to for every case but a new frame). don't recalculate it for an annihilation scan.

dankamongmen commented 3 years ago

Alright, this sixel plan is slowly coming together. I've written extract_palette(), and it seems to work. I've modified write_sixel_data() to work without a deets table, which it'll not have available in the update case (and never will, if we change p the quantization). Need to now write a function to build up the data table via matching...yuck what a pain in the ass, maybe doing it inline is the best way, since we don't have the original bitmap handy. ugh!

dankamongmen commented 3 years ago

I think the sixelmap idea might be taking us too far away from the sixel glyph. Rather than keep a pure index matrix (which is expensive to bring back to a Sixel), and rather than performing delicate, difficult, expensive operations directly on the glyph form, let's keep around the (color-indexed) data array of sixels themselves. We can index into these in O(1), just like kitty. We can even probably get rid of the weird sixel_update() function, and work via invalidation, reconstructing based on a flag.

dankamongmen commented 3 years ago

I think this will work well, but let's make sure we shrink down data prior to saving it, as it's currently per-color-register, and we only need per-color.

dankamongmen commented 3 years ago

ugh, just saw a segfault in kitty's restoration code, reproducible, use balloon.png from rgb:

==2037358==    at 0x4B19CBF: raise (raise.c:46)
==2037358==    by 0x4B19D5F: ??? (in /lib/x86_64-linux-gnu/libc-2.31.so)
==2037358==    by 0x486F401: kitty_restore (kitty.c:213)
==2037358==    by 0x486F401: kitty_rebuild (kitty.c:306)
==2037358==    by 0x4884452: sprite_rebuild (internal.h:1006)
==2037358==    by 0x4884452: paint_sprixel (render.c:186)
==2037358==    by 0x4884452: paint (render.c:236)
==2037358==    by 0x4886C4D: ncpile_render_internal (render.c:1287)
==2037358==    by 0x4886C4D: ncpile_render (render.c:1359)
==2037358==    by 0x4887870: notcurses_render (render.c:1373)
==2037358==    by 0x10C0B2: render (NotCurses.hh:200)
==2037358==    by 0x10C0B2: perframe(ncvisual*, ncvisual_options*, timespec const*, void*) (play.cpp:121)
==2037358==    by 0x4851A04: ffmpeg_stream (ffmpeg.c:425)
==2037358==    by 0x10C9A9: stream (Visual.hh:76)
==2037358==    by 0x10C9A9: rendered_mode_player_inner(ncpp::NotCurses&, int, char**, ncscale_e, ncblitter_e, bool, bool, double, double, unsigned int) (play.cpp:409)
==2037358==    by 0x10CDFA: rendered_mode_player(int, char**, ncscale_e, ncblitter_e, notcurses_options&, bool, bool, double, double, unsigned int) (play.cpp:478)
==2037358==    by 0x10BAB8: main (play.cpp:515)
==2037358== 
==2037358== HEAP SUMMARY:

dankamongmen commented 3 years ago

ugh, just saw a segfault in kitty's restoration code, reproducible, use balloon.png from rgb:

you have to do two files to get this, btw, like:

[schwarzgerat](0) $ valgrind --tool=memcheck  ./ncplayer -bpixel /media/chungus/images/allrgb/balloon.png /media/chungus/images/allrgb/balloon.png 2>e

 notcurses 2.2.8 by nick black et al on xterm-kitty
  70 rows (20px) 80 cols (10px) (87.50KiB) 48B crend 256 colors+RGB
  compiled with gcc-10.2.1 20210110, 16B little-endian cells
  terminfo from ncurses 6.2.20201114
  avformat 58.79.100 avutil 56.74.100 swscale 5.10.100
Segmentation fault
[schwarzgerat](139) $ cat e
==2041725== Memcheck, a memory error detector
==2041725== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2041725== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==2041725== Command: ./ncplayer -bpixel /media/chungus/images/allrgb/balloon.png /media/chungus/images/allrgb/balloon.png
==2041725== 
==2041725== Invalid read of size 1
==2041725==    at 0x486F402: kitty_restore (kitty.c:216)
==2041725==    by 0x486F402: kitty_rebuild (kitty.c:306)
==2041725==    by 0x4884452: sprite_rebuild (internal.h:1006)
==2041725==    by 0x4884452: paint_sprixel (render.c:186)
==2041725==    by 0x4884452: paint (render.c:236)
==2041725==    by 0x4886C4D: ncpile_render_internal (render.c:1287)
==2041725==    by 0x4886C4D: ncpile_render (render.c:1359)
==2041725==    by 0x4887870: notcurses_render (render.c:1373)
==2041725==    by 0x10C0B2: render (NotCurses.hh:200)
==2041725==    by 0x10C0B2: perframe(ncvisual*, ncvisual_options*, timespec const*, void*) (play.cpp:121)
==2041725==    by 0x4851A04: ffmpeg_stream (ffmpeg.c:425)
==2041725==    by 0x10C9A9: stream (Visual.hh:76)
==2041725==    by 0x10C9A9: rendered_mode_player_inner(ncpp::NotCurses&, int, char**, ncscale_e, ncblitter_e, bool, bool, double, double, unsigned int) (play.cpp:409)
==2041725==    by 0x10CDFA: rendered_mode_player(int, char**, ncscale_e, ncblitter_e, notcurses_options&, bool, bool, double, double, unsigned int) (play.cpp:478)
==2041725==    by 0x10BAB8: main (play.cpp:515)
==2041725==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==2041725== 
==2041725== 
==2041725== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==2041725==    at 0x4B19CBF: raise (raise.c:46)
==2041725==    by 0x4B19D5F: ??? (in /lib/x86_64-linux-gnu/libc-2.31.so)
==2041725==    by 0x486F401: kitty_restore (kitty.c:213)
==2041725==    by 0x486F401: kitty_rebuild (kitty.c:306)
==2041725==    by 0x4884452: sprite_rebuild (internal.h:1006)
==2041725==    by 0x4884452: paint_sprixel (render.c:186)
==2041725==    by 0x4884452: paint (render.c:236)
==2041725==    by 0x4886C4D: ncpile_render_internal (render.c:1287)
==2041725==    by 0x4886C4D: ncpile_render (render.c:1359)
==2041725==    by 0x4887870: notcurses_render (render.c:1373)
==2041725==    by 0x10C0B2: render (NotCurses.hh:200)
==2041725==    by 0x10C0B2: perframe(ncvisual*, ncvisual_options*, timespec const*, void*) (play.cpp:121)
==2041725==    by 0x4851A04: ffmpeg_stream (ffmpeg.c:425)
==2041725==    by 0x10C9A9: stream (Visual.hh:76)
==2041725==    by 0x10C9A9: rendered_mode_player_inner(ncpp::NotCurses&, int, char**, ncscale_e, ncblitter_e, bool, bool, double, double, unsigned int) (play.cpp:409)
==2041725==    by 0x10CDFA: rendered_mode_player(int, char**, ncscale_e, ncblitter_e, notcurses_options&, bool, bool, double, double, unsigned int) (play.cpp:478)
==2041725==    by 0x10BAB8: main (play.cpp:515)
==2041725== 
==2041725== HEAP SUMMARY:
==2041725==     in use at exit: 114,256,625 bytes in 1,399 blocks
==2041725==   total heap usage: 103,759 allocs, 102,360 frees, 3,058,423,429 bytes allocated
==2041725== 
==2041725== LEAK SUMMARY:
==2041725==    definitely lost: 0 bytes in 0 blocks
==2041725==    indirectly lost: 0 bytes in 0 blocks
==2041725==      possibly lost: 1,607 bytes in 21 blocks
==2041725==    still reachable: 114,255,018 bytes in 1,378 blocks
==2041725==                       of which reachable via heuristic:
==2041725==                         newarray           : 1,536 bytes in 16 blocks
==2041725==         suppressed: 0 bytes in 0 blocks
==2041725== Rerun with --leak-check=full to see details of leaked memory
==2041725== 
==2041725== For lists of detected and suppressed errors, rerun with: -s
==2041725== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
[schwarzgerat](0) $

dankamongmen commented 3 years ago

ok, got the kitty issue fixed, whew

dankamongmen commented 3 years ago

Alright, I've got sixel wiping working again finally. Seems to work well. Now to rebuild.

dankamongmen commented 3 years ago

we've got the sixel auxvec cycle in now. not working perfectly, but the skeleton is there.

dankamongmen commented 3 years ago

sixel wipe+rebuild almost works. we're hitting 1 cell's worth too far down somehow in sixel_wipe(). on natasha-blur.png, the wiped section begins at cell 18, on a 23px-tall cell, aka y == 414. wipe_sixel() is instead using 437, which is of course 414 + 23. that's fucking us up, and also the cause of our lingering valgrind warning. i've been staring at this for an hour and am unsure why this is happening -- going to take a break and come back to it. by the end of the night, we will have working sixel wipe+rebuild.

dankamongmen commented 3 years ago

so very close on sixel rebuild! just a pixel or two off

2021-04-28-232531_884x556_scrot

dankamongmen commented 3 years ago

a bit closer...

2021-04-28-233915_884x556_scrot

dankamongmen commented 3 years ago

top is now perfectly reconstructed, but we run into problems wiping towards the bottom. so close!

2021-04-28-234124_884x556_scrot

dankamongmen commented 3 years ago

oh, how i want to be done with this shitty bug

dankamongmen commented 3 years ago

before i commit this, i need to add the transvec for sixel. right now i'm just capping the number of colors to 255, so i can use one as a sentinel. add the actual transvec, and reclaim our lost color register.

dankamongmen commented 3 years ago

we're still seeing a few valgrind warnings. here's intro in an xterm with geometry rows: 61 cols: 147 rpx: 23 cpx: 11 (1403x1617):

smap->data[2677397] = 00 boff: 2677200 x: 197 color: 104                        
band: 85 color: 104 idx: 5329687 mask: 3e                                       
==702877== Invalid read of size 1                                               
==702877==    at 0x4A18336: wipe_color (sixel.c:806)                            
==702877==    by 0x4A18336: sixel_wipe (sixel.c:857)                            
==702877==    by 0x4A18B0A: sprite_wipe (sprite.c:186)                          
==702877==    by 0x4A103A4: paint_sprixel (render.c:172)                        
==702877==    by 0x4A103A4: paint (render.c:236)                                
==702877==    by 0x4A12C7D: ncpile_render_internal (render.c:1287)              
==702877==    by 0x4A12C7D: ncpile_render (render.c:1359)                       
==702877==    by 0x4A138A0: notcurses_render (render.c:1373)                    
==702877==    by 0x117687: demo_render.part.0 (hud.c:623)                       
==702877==    by 0x119F7F: orcaride (intro.c:103)                               
==702877==    by 0x119F7F: intro (intro.c:245)                                  
==702877==    by 0x11057B: ext_demos (demo.c:225)                               
==702877==    by 0x11057B: main (demo.c:583)                                    
==702877==  Address 0x10601bf7 is 119 bytes inside a block of size 8,192 free'd 
==702877==    at 0x48399AB: free (vg_replace_malloc.c:538)                      
==702877==    by 0x4A08942: ncplane_vprintf_yx (notcurses.c:1690)               
==702877==    by 0x4A08F4C: ncplane_printf_yx (notcurses.h:1754)                
==702877==    by 0x4A0970F: redraw_plot_uint64_t (plot.c:415)                   
==702877==    by 0x1174AE: demo_render.part.0 (hud.c:589)                       
==702877==    by 0x119ED8: animate (intro.c:25)                                 
==702877==    by 0x119ED8: intro (intro.c:235)                                  
==702877==    by 0x11057B: ext_demos (demo.c:225)                               
==702877==    by 0x11057B: main (demo.c:583)                                    
==702877==  Block was alloc'd at                                                
==702877==    at 0x483877F: malloc (vg_replace_malloc.c:307)                    
==702877==    by 0x4A0219A: ncplane_vprintf_prep (notcurses.c:1658)             
==702877==    by 0x4A0891E: ncplane_vprintf_yx (notcurses.c:1685)               
==702877==    by 0x4A08F4C: ncplane_printf_yx (notcurses.h:1754)                
==702877==    by 0x4A0970F: redraw_plot_uint64_t (plot.c:415)                   
==702877==    by 0x1174AE: demo_render.part.0 (hud.c:589)                       
==702877==    by 0x119ED8: animate (intro.c:25)                                 
==702877==    by 0x119ED8: intro (intro.c:235)                                  
==702877==    by 0x11057B: ext_demos (demo.c:225)                               
==702877==    by 0x11057B: main (demo.c:583)                                    
==702877==            
              smap->data[2677687] = 00 boff: 2677500 x: 187 color: 104                        
==702877== Invalid read of size 1                                               
==702877==    at 0x4A18376: wipe_color (sixel.c:810)                            
==702877==    by 0x4A18376: sixel_wipe (sixel.c:857)                            
==702877==    by 0x4A18B0A: sprite_wipe (sprite.c:186)                          
==702877==    by 0x4A103A4: paint_sprixel (render.c:172)                        
==702877==    by 0x4A103A4: paint (render.c:236)                                
==702877==    by 0x4A12C7D: ncpile_render_internal (render.c:1287)              
==702877==    by 0x4A12C7D: ncpile_render (render.c:1359)                       
==702877==    by 0x4A138A0: notcurses_render (render.c:1373)                    
==702877==    by 0x117687: demo_render.part.0 (hud.c:623)                       
==702877==    by 0x119F7F: orcaride (intro.c:103)                               
==702877==    by 0x119F7F: intro (intro.c:245)                                  
==702877==    by 0x11057B: ext_demos (demo.c:225)                               
==702877==    by 0x11057B: main (demo.c:583)                                    
==702877==  Address 0x10601bf7 is 119 bytes inside a block of size 8,192 free'd 
==702877==    at 0x48399AB: free (vg_replace_malloc.c:538)                      
==702877==    by 0x4A08942: ncplane_vprintf_yx (notcurses.c:1690)               
==702877==    by 0x4A08F4C: ncplane_printf_yx (notcurses.h:1754)                
==702877==    by 0x4A0970F: redraw_plot_uint64_t (plot.c:415)                   
==702877==    by 0x1174AE: demo_render.part.0 (hud.c:589)                       
==702877==    by 0x119ED8: animate (intro.c:25)                                 
==702877==    by 0x119ED8: intro (intro.c:235)                                  
==702877==    by 0x11057B: ext_demos (demo.c:225)                               
==702877==    by 0x11057B: main (demo.c:583)                                    
==702877==  Block was alloc'd at                                                
==702877==    at 0x483877F: malloc (vg_replace_malloc.c:307)                    
==702877==    by 0x4A0219A: ncplane_vprintf_prep (notcurses.c:1658)             
==702877==    by 0x4A0891E: ncplane_vprintf_yx (notcurses.c:1685)               
==702877==    by 0x4A08F4C: ncplane_printf_yx (notcurses.h:1754)                
==702877==    by 0x4A0970F: redraw_plot_uint64_t (plot.c:415)                   
==702877==    by 0x1174AE: demo_render.part.0 (hud.c:589)                       
==702877==    by 0x119ED8: animate (intro.c:25)                                 
==702877==    by 0x119ED8: intro (intro.c:235)                                  
==702877==    by 0x11057B: ext_demos (demo.c:225)                               
==702877==    by 0x11057B: main (demo.c:583)                                    
==702877==                                                                      
band: 85 color: 104 idx: 5329688 mask: 3e                                       
smap->data[2677688] = 00 boff: 2677500 x: 188 color: 104                        
band: 85 color: 104 idx: 5329689 mask: 3e                                       
smap->data[2677689] = 00 boff: 2677500 x: 189 color: 104

dankamongmen commented 3 years ago

this from above is topped by: didx: 104 sixels: 25500 color: 104 B: 84-85 Y: 506-510 X: 187-197 coff: 2652000,

dankamongmen commented 3 years ago

There ought only be 2677500 data slots (105 * 25500), but we're reading from 2677687 here. The difference would be 187 (on a motherfuckin' cop).

dankamongmen commented 3 years ago

> 104 * 25500 + 510/6*300 + 197
2677697
>

hrmmm.

dankamongmen commented 3 years ago

we need 86 * 300 sixels, aka 25800. then this all works.

dankamongmen commented 3 years ago

no, we only need 85. 510 / 6 == 85. that ought be enough....no. we're handling Ys 506..510. that's lines 0..510 which is 511 lines. so is that the problem? we shouldn't actually be handling row 510 (which is actually the 511th row)? we cut the sixel to a multiple of 6...yeah, we oughtn't be handling row 510. stupid.

dankamongmen commented 3 years ago

i think we got it =]. 0331 2021-04-29.

dankamongmen commented 3 years ago

yep! sixel wipe/rebuild now works perfectly, so far as i can tell =] =] =].

dankamongmen commented 3 years ago

hrmm, rebuilding the auxvectors in sixel is going to be tricky, since we don't have the palette until quantization is done, but need the palette to fill in the auxvec. erp. not sure exactly what to do there. this code extracts it as we go along, but that's no good:

[schwarzgerat](0) $ git diff
diff --git src/lib/sixel.c src/lib/sixel.c
index f5fd66782..4931e4229 100644
--- src/lib/sixel.c
+++ src/lib/sixel.c
@@ -261,11 +261,41 @@ extract_color_table(const uint32_t* data, int linesize, int cols,
       for(int sy = visy ; sy < (begy + leny) && sy < visy + 6 ; ++sy){ // offset within sprixel
         const uint32_t* rgb = (data + (linesize / 4 * sy) + visx);
         int txyidx = (sy / cdimy) * cols + (visx / cdimx);
-        if(tam[txyidx].state == SPRIXCELL_ANNIHILATED || tam[txyidx].state == SPRIXCELL_ANNIHILATED_TRANS){
+        int c; // negative means transparent
+        if(rgba_trans_p(*rgb, bargs->transcolor)){
+          c = -1;
+        }else{
+          unsigned char comps[RGBSIZE];
+          break_sixel_comps(comps, *rgb, mask);
+          if((c = find_color(stab, comps)) < 0){
+//fprintf(stderr, "FAILED FINDING COLOR AUGH 0x%02x\n", mask);
+            return -1;
+          }
+        }
+        // if we're SPRIXCELL_ANNIHILATED, we need rebuild the auxvec. if there
+        // are no opaque pixels, we transition to SPRIXCELL_ANNIHILATED_TRANS.
+        // if we're SPRIXCELL_ANNIHILATED_TRANS, and there are any opaque
+        // pixels, we transition to SPRIXCELL_ANNIHILATED and load the auxvec.
+        // otherwise, we're valid data, and ought set the correct state.
+        uint8_t* auxvec = tam[txyidx].auxvector;
+        // we only have an auxvec if we're some form of transparent.
+        if(auxvec){
 //fprintf(stderr, "TRANS SKIP %d %d %d %d (cell: %d %d)\n", visy, visx, sy, txyidx, sy / cdimy, visx / cdimx);
+          if(tam[txyidx].state == SPRIXCELL_ANNIHILATED_TRANS){
+            if(c >= 0){
+              tam[txyidx].state = SPRIXCELL_ANNIHILATED;
+            }
+          }
+          // FIXME need to transition to ANNIHILATED_TRANS with no opaques
+          int auxidx = (sy % cdimy) * cdimx + (visx % cdimx);
+          // FIXME need a real transparency matrix
+          if(c >= 0){
+fprintf(stderr, "updating %d %d -> %d\n", auxidx, auxvec[auxidx], c);
+            auxvec[auxidx] = c;
+          }
           continue;
         }
-        if(rgba_trans_p(*rgb, bargs->transcolor)){
+        if(c < 0){
           if(sy % cdimy == 0 && visx % cdimx == 0){
             tam[txyidx].state = SPRIXCELL_TRANSPARENT;
           }else if(tam[txyidx].state == SPRIXCELL_OPAQUE_SIXEL){
@@ -280,13 +310,6 @@ extract_color_table(const uint32_t* data, int linesize, int cols,
             tam[txyidx].state = SPRIXCELL_MIXED_SIXEL;
           }
         }
-        unsigned char comps[RGBSIZE];
-        break_sixel_comps(comps, *rgb, mask);
-        int c = find_color(stab, comps);
-        if(c < 0){
-//fprintf(stderr, "FAILED FINDING COLOR AUGH 0x%02x\n", mask);
-          return -1;
-        }
         stab->map->data[c * stab->map->sixelcount + pos] |= (1u << (sy - visy));
         update_deets(*rgb, &stab->deets[c]);
 //fprintf(stderr, "color %d pos %d: 0x%x\n", c, pos, stab->data[c * stab->map->sixelcount + pos]);
[schwarzgerat](0) $

dankamongmen commented 3 years ago

i think i'm gonna add the transparency vector to sixel, and then merge. we'll handle updating the auxvectors in another issue. i'm done with this one.

dankamongmen / notcurses

need restore bitmaps when no longer occluded #1440