Sixel background select when scrolling

[j4james]

This is an issue that's been raised a couple of times since Windows Terminal added sixel support. What happens when you specify a background select area that extends beyond the bottom of the screen, and then output sixel content that triggers a scroll? How much of the screen is ultimately filled by that background select?

My assumption is that the background select is a distinct operation that happens at the start, and is clamped to the boundaries of the screen. If you output enough sixels to force the screen to scroll, it's not going to fill any more of the background at that point, because it's already done with the background select.

I'm fairly confident that my understanding is correct, but I've been wrong before, so it would be good to get some confirmation of the correct behavior. So if you ever get back to testing on your VT340 again, I'd appreciate you giving this test script a run:

#!/bin/bash

# Test the effect of Background Select when an image scrolls.

CSI=$'\e['          # Control Sequence Introducer 
DCS=$'\eP'          # Device Control String
ST=$'\e\\'          # String Terminator

echo -n ${CSI}'!p'
echo -n ${CSI}'H'
echo -n ${CSI}'J'

# Enable reverse screen mode so the background is white
echo -n ${CSI}'?5h'

# Position the cursor 3 rows from the bottom
echo -n ${CSI}'22;35H'

# Fill 120x120 of the background in black (clamped to 3 rows, 60 pixels)
echo -n ${DCS}'q"1;1;120;120'

# Render a 60x60 square in blue on the bottom 3 rows
echo -n '#1!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~-'

# Render a 60x60 square in red, scrolling up another 3 rows
echo -n '#2!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~'

# Apply a bunch of graphic new lines to scroll the image up to the middle
echo -n '------------------------------'

# End the image sequence
echo -n ${ST}

# Move the cursor back to home
echo -n ${CSI}'H'

If my understanding is correct, this should render three equally sized squares in an r pattern like this:

If I'm wrong, the bottom right corner of that pattern may be filled in black as well.

[hackerb9]

I am amidst the whirlwind, but have touched down briefly at home. I will try to test today.

[hackerb9]

Seems it is slightly more complex than expected.

I haven't investigated, but offhand this looks like something that is due to the VT340's peculiar interaction between foreground and background text with the sixel color palette and may not be something that any modern terminal emulator will want to replicate.

[j4james]

Wow! That was not what I was expecting at all. But thanks for the quick test. I'll have to see how easy that is to replicate. My current implementation is definitely not what users are expecting, so if it's not technically correct, there's no justification for leaving it as it is.

[hackerb9]

Background color switching on the VT340 is indirect; it does not change color 0 (text background) in the colormap. Since the text color and graphics colormap are shared on the VT340, the sixel image's background is not going to look correct.

I think it'd be justified to omit this feature on any terminal which does not tie the text colormap to the sixel colormap. (I believe your Microsoft implementation decided to not do things that way, right?)

[hackerb9]

I realized, you can almost get the image you had expected by doing something silly. Change the line which draws the blue square to:

# Render a 60x60 square in blue on the bottom 3 rows, omit final newline        
echo -n '#1!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~'
# Paint background as a 1x1 black pixel, send newline. (Kludge-o-rama!)         
echo -n '";;1;1-'

That results in:

[j4james]

I get that the graphics background isn't the same as the text, but that's exactly the problem modern terminals have. If they shared the same color table (as they do on the VT340), it's unlikely anyone would even notice the effect of this (you'd have to be using reverse screen mode like I did in the test above). But modern terminals can have weird color schemes, where the text background can be quite different from the default black of the graphics palette, so when we get this wrong it's immediately obvious.

As a common example, a transparent gif in img2sixel is rendered with the background filled, and then only the non-transparent pixels drawn on top of that. If you output that at the bottom of the screen, we only do the background fill for one row, because that's the maximum area it can cover at the point the raster attributes are interpreted. But then the remaining lines of the image don't have a background.

So you get something like this:

The first image was output in the middle of the screen, but second image was at the bottom, and triggered a scroll. If I matched the VT340 behavior correctly, those should both look the same I think.

[j4james]

The thing I'm not sure about now is how far the VT340 does actually fill in these circumstance. It's clearly not paying any attention to the specified raster attributes height, but does that mean it fills the full height of any text row that scrolls when it's active? Or is it only the height of the sixel rows that have been output? If it's the former, then we're still going to end up with situations where a scrolled image looks different from one that didn't scroll.

I need to go to bed and think about this some more tomorrow.

[j4james]

I've got my implementation working with the two examples above, but there are so many edge cases that I'm not at all confident that what I'm doing is correct. So I've now made an extended version of the test which will hopefully reveal some more details of the VT340's implementation. If I've managed to match it on this test, I'll consider my code good enough to ship.

#!/bin/bash

# Test the effect of Background Select when an image scrolls.

CSI=$'\e['          # Control Sequence Introducer 
DCS=$'\eP'          # Device Control String
ST=$'\e\\'          # String Terminator

echo -n ${CSI}'!p'
echo -n ${CSI}'H'
echo -n ${CSI}'J'

# Enable reverse screen mode so the background is white
echo -n ${CSI}'?5h'

# Background select with varying width and aspect ratio
echo -n ${CSI}'19;20H'
echo -n ${DCS}'q"20;1;240;240'
echo -n '#1!120~'
echo -n '"10;1;120;1--'
echo -n '#2!60~'
echo -n '"5;1;60;1--'
echo -n '#3!30~'
echo -n ${ST}
echo -en '\b\b>\n\n'

# Background height exactly fits, and is a multiple of 6.
echo -n ${CSI}'22;37H'
echo -n ${DCS}'q"1;1;120;60'
echo -n '#1!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~-!60~----'
echo -n ${ST}
echo -en '\b\b>'

# Background height extends beyond the bottom margin.
echo -n ${CSI}'24;50H'
echo -n ${DCS}'q"1;1;60;21'
echo -n '#2!30~-!30~-!30~-!30~-!30~'
echo -n ${ST}
echo -en '\b\b>\n'

# Background height exactly fits, but isn't a multiple of 6.
echo -n ${CSI}'24;58H'
echo -n ${DCS}'q"1;1;40;20'
echo -n '#3!20~-!20~-!20~-!20B----------'
echo -n ${ST}
echo -en '\b\b>'

# Move the cursor back to home
echo -n ${CSI}'H'

[j4james]

For reference, this is what I'm hoping the above test will look like:

1. The first test is looking at the effects of the raster attributes width and aspect ratio on the background fill when scrolling. The part I'm most unsure of is the last block, which scrolls a sixel band that isn't a multiple of the row height. If the background is filled with a multiple of the row height (rather than the sixel band height), the black part won't align with the green.

2. The second test is checking whether the background continues to be filled when the image scrolls if the initial raster height fits on screen. The image is output with a height of 60px when there is exactly enough space for 60px, but then adds a few more graphic new lines to force a scroll. I'm assuming that additional area is not filled, but if it is, the black part won't align with the blue.

3. The third test is checking how much of the background is filled when the initial raster height doesn't fit on screen. The image is output with a height of 21px, when there is only enough space for 20px. A 30x30px block is then output which triggers a scroll. I'm assuming that will result in an additional 10px of the background being filled. But if it always fills a multiple of the row height (i.e. 20px), the black part won't align with the red.

4. The fourth test is similar to the second, checking that the background doesn't fill when scrolling if the initial raster height fits on screen. In this case, though, the initial height (20px) is not a multiple of 6, which might make a difference to the result.

The > characters are just showing where the cursor position ends up after each test. I'm not anticipating any unexpected behavior there, but it's possible this'll uncover some new complication I wasn't previously aware of.