How does the background select really work?

[j4james]

From the results we've got so far in the margin_clipping and raster_dimensions tests, I know there are some aspects of the background select that I've got wrong, but I'm still not completely sure what the correct behavior is. So before I make any more adjustments to those tests, I'll like to try and get a better understanding of the edge cases.

To start with, it looks to me like the VT340 only starts filling the background once you've output at least one pixel. So my first question is, what happens if the sequence didn't include any sixels at all? Would it just have no effect and not output anything, i.e. in these two tests, would you just get a screen that is filled with E's?

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq\033\\n' printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400;240\033\\n'

And if that's the case, what does is take for the background select to kick in? Is a 'zero' sixel enough (i.e. ?), or would you need at least one non-zero sixel (e.g. @)? Do both of these tests fill the screen with black?

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq?\033\\n' printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq#0@\033\\n'

And testing with an explicit raster size, do both of these fill the top left quarter in black?

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400;240?\033\\n' printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400;240#0@\033\\n'

Another thing implied by the current results is that a graphic new line wouldn't be enough to trigger the start of the background fill. If that's correct, then I'd expect only the bottom half of the screen to be filled with black in these next two cases. Is that correct?

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033P2q--------?\033\\n' printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033P2q--------#0@\033\\n'

And testing with an explicit raster size, I'd expect these two to fill the middle left portion of the screen (rows 6 to 17, columns 0 to 39).

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"5;1;400;240----?\033\\n' printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"5;1;400;240----#0@\033\\n'

Another area I'm unsure of is the handling of the zero and omitted parameters in the raster size definition. Contrary to what the DEC STD-070 documentation says, my understanding is that both zero and omitted parameters are treated as representing the maximum extent (rather than 1). If that is correct, then the two examples below should fill the top half of the screen (given an explicit height, but default width).

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;;240?\033\\n' printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;0;240?\033\\n'

And these two should fill the left half of the screen (given an explicit width, but default height).

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400?\033\\n' printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400;0?\033\\n'

I'm not expecting you to screenshot all of these - I'm just looking for confirmation whether they are or aren't filling the screen in the way I expect. Once I think I know what's going on, I'll try and update my main test cases to take these results into account.

[j4james]

It just occurred to me that you can't exactly cut and paste these examples into the VT340. I'm assuming it'll be easier to test with a simple shell script that runs them one by one.

#!/bin/bash

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq\033\\\n'
printf ' 1. Screen filled with Es? '
read -sn 1
printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400;240\033\\\n'
printf ' 2. Screen filled with Es? '
read -sn 1

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq?\033\\\n'
printf ' 3. Screen completely black? '
read -sn 1
printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq#0@\033\\\n'
printf ' 4. Screen completely black? '
read -sn 1

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400;240?\033\\\n'
printf ' 5. Top left quarter black? '
read -sn 1
printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400;240#0@\033\\\n'
printf ' 6. Top left quarter black? '
read -sn 1

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033P2q--------?\033\\\n'
printf ' 7. Bottom half black? '
read -sn 1
printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033P2q--------#0@\033\\\n'
printf ' 8. Bottom half black? '
read -sn 1

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"5;1;400;240----?\033\\\n'
printf ' 9. Middle left black? '
read -sn 1
printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"5;1;400;240----#0@\033\\\n'
printf ' 10. Middle left black? '
read -sn 1

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;;240?\033\\\n'
printf ' 11. Top half black? '
read -sn 1
printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;0;240?\033\\\n'
printf ' 12. Top half black? '
read -sn 1

printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400?\033\\\n'
printf ' 13. Left half black? '
read -sn 1
printf '\033[H\033[?7h';yes E|tr -d '\n'|head -c 1920;printf '\033[H\033Pq"1;1;400;0?\033\\\n'
printf ' 14. Left half black? '
read -sn 1

[hackerb9]

The github website gave me a textarea form I could fill in that I thought was for the response, but on the VT340 I couldn't tell I was actually editing your comment! Hah! ☺ Anyhow, the answer is "No" to the first two and "Yes" to all the rest.

[j4james]

Thanks for testing that. That makes sense. I guess it waits for either the first sixel to be ouput, or the end of the image.

[j4james]

Although now that I think about it, that doesn't explain the second test. If it was going to fill anything, I would have expected it to be limited to the 400x240 size, i.e. a quarter of the screen. So that is a bit strange.

[hackerb9]

I put a sleep in the first test and it seems to clear the screen when the first byte after q is received. (I didn't send the backslash for five seconds). It's possible there is some byte which it would not clear the screen for, but I'm not sure what.

[hackerb9]

Oh, duh. Of course, it's waiting for DECGRA. So a double quote after the q does not clear the screen.

[hackerb9]

For the second test, it clears the screen unless there is some sixel data. For example, this clears the top left corner (as soon as it receives the escape character):

printf '\e[H\ePq"1;1;400;240#0'; sleep 5; printf '\e\\'

This clears the upperleft corner when it receives the letter 'a':

printf '\e[H\ePq"1;1;400;240#0a'; sleep 5; printf '\e\\'

[hackerb9]

I think clearing the whole screen when there is no sixel data is a bug as the following should be equivalent, but correctly clears just the top left corner:

printf '\e[H\ePq"1;1;400;240"1;1;400;240'; sleep 5; printf '\e\\'

[UPDATE: new test cases show that the second DECGRA is ignored unless there is sixel data]

This clears a 400x240 square:

printf '\e[H\ePq"1;1;400;240"1;1;200;120'; sleep 5; printf '\e\\'

This clears a 200x120 square:

printf '\e[H\ePq"1;1;400;240"1;1;200;120'"1;1;100;60'; sleep 5; printf '\e\\'

This clears a 100x60 square:

printf '\e[H\ePq"1;1;400;240"1;1;200;120'"1;1;100;60#'; sleep 5; printf '\e\\'

I'm not sure what makes it sometimes work . A single hashmark (with no color number) works, which rules out my theory that the entire screen was clearing as a sign that the sixel data was malformed. Semicolons and numbers don't, so perhaps it is simply any token that takes the VT340's finite state machine out of the initial "getting sixel metadata" state.

[j4james]

This is getting really interesting. It didn't even occur to me until now that you could set DECGRA more than once.

Semicolons and numbers don't, so perhaps it is simply any token that takes the VT340's finite state machine out of the initial "getting sixel metadata" state.

Yeah, that seems like a reasonable theory.

[hackerb9]

Speaking of weird finite state machine things, maybe I'm just tired, but I don't understand how my VT340 seems to know to get out of sixel mode when I get back to the prompt when I don't send the String Terminator.

# Gives me a prompt on the VT340, xterm; gets stuck on mlterm
printf '\e[H\ePq"1;1;400;240' 

[time passes...]

Okay, I just put the VT340 into "Show Control Characters" mode (instead of "Interpret") and I can see that my shell (bash) is sending Esc [ ? 20004 h before the prompt I had set (PS1="$ "). That's the special "bracketed paste mode" invented by xterm. Apparently, both xterm and a true VT340 abort out of sixel mode (perhaps discarding graphics not drawn, yet?) when they receive the CSI (Control Sequence Introducer, Esc [).

This seems like an important quirk to test and document for VT340 sixel compatibility. It actually makes experimenting with sixels much nicer because I haven't been getting stuck in sixel mode with no clear visual indication of what's going on.

[j4james]

any token that takes the VT340's finite state machine out of the initial "getting sixel metadata" state.

I've been playing around with this on my own Sixel implementation, and the VT340 behavior seems reasonably sensible in retrospect. Sixel commands don't have a specific separator indicating where the command ends, so you generally execute a pending command when you encounter the start of the next command, or when you receive a sixel byte. Originally I would also execute any pending commands when the end of the image was reached (i.e. the ESC of the ST). But if I drop that final execute step, my behavior now exactly matches what you're seeing on the VT340.

There is a catch, though. I don't think this rule applies to DECGCI. If there's a pending palette change at the time the image is terminated, I think that would still be executed. For example, your original resetpalette script was just a bunch of DECGCI commands, and the last DECGCI wasn't followed by any other commands or sixel data. Yet that final command was still being executed (or so I assume - I think you would have noticed if the background color wasn't reset).

Apparently, both xterm and a true VT340 abort out of sixel mode (perhaps discarding graphics not drawn, yet?) when they receive the CSI (Control Sequence Introducer, Esc [).

Technically I think the ANSI/ECMA specs require that a DCS sequence be terminated by an ST, and thus shouldn't be executed if it isn't. But as far as I know, the DEC terminals treat any ESC, CAN, SUB, or C1 control as a valid terminator. This is mentioned in 3.5.4.4 of the STD-070 manual, and I know the VT240 worked that way, so I assumed the VT340 did as well. And in the case of Sixel, the image is already being output as it's received, so aborting doesn't make much sense.

That said, none of the open source terminal emulators handle progressive output (at least that I'm aware of), and many of them do actually abort the image if it's not terminated with an ST (i.e. they don't display anything at all). But that's a bug as far as I'm concerned. There's going to be software that just won't work, because it's dependent on the behavior of the original DEC terminals. I know I've encountered old Sixel images and DECDLD fonts that aren't "correctly" terminated.

This seems like an important quirk to test and document for VT340 sixel compatibility.

Yeah, it's definitely worth testing, but I didn't think there was much doubt about the behavior, and initially I've been concentrating on testing the edge cases for which there wasn't any clear documentation. Then again, I've frequently been wrong about VT340 behavior which I thought I understood perfectly.

[hackerb9]

Technically I think the ANSI/ECMA specs require that a DCS sequence be terminated by an ST, and thus shouldn't be executed if it isn't. But as far as I know, the DEC terminals treat any ESC, CAN, SUB, or C1 control as a valid terminator. This is mentioned in 3.5.4.4 of the STD-070 manual, and I know the VT240 worked that way, so I assumed the VT340 did as well. And in the case of Sixel, the image is already being output as it's received, so aborting doesn't make much sense.

I hadn't known that those characters could stand in for ST. Yet another good test to add to the list for any future sixel implementations.

[j4james]

FYI, I've included a test for different string terminators in PR #8. That test also checks what happens when you embed other control characters and unexpected command/parameter characters in the control string.

[hackerb9]

Results are in unexpected.png. Note that the backwards question mark is the VT340 way of indicating an unknown character.

[j4james]

Note that the backwards question mark is the VT340 way of indicating an unknown character.

Yeah, I was expecting the reverse question mark. The normal question mark at the end of the test pattern was to try and balance that out (the idea being that it should look reasonably symmetric if it was working correctly). I wasn't expecting it to be quite that large, but that's not the end of the world.

I am surprised that all the squares are gray though. They were meant to be kind of chequered pattern with various shades of yellow and gray. Is it possible you ran the test when the palette was set to monochrome? If the test had actually "failed", I would expected more corruption of the shape, and at least some sign of color.

This is what I was expecting:

[hackerb9]

Bizarrely, that is exactly what displays on the screen. I tried it again and, during MediaCopy command, most of the colors got switched to 47;47;47. Here's is the header of the resulting sixel file:

ESC[2 ESCP0;1;6q"1;1;800;480
#1;2;47;47;47
#2;2;47;47;47
#3;2;47;47;47
#4;2;47;47;47
#5;2;47;47;47
#6;2;47;47;47
#7;2;47;47;47
#8;2;47;47;47
#9;2;47;47;47
#10;2;47;47;47
#11;2;47;47;47
#12;2;47;47;47
#13;2;47;47;47
#14;2;47;47;47
#15;2;80;80;80

If that palette is replaced with the default one, the image shows up exactly as you expected.

I double-checked and my mediacopy.sh script is requesting a color print. I did notice I had left it creating a transparent image ("no background"), but that shouldn't have affected the palette. I'm rerunning the script and will let you know if I figure out where the glitch is.

[hackerb9]

No idea what went wrong before. This time it worked. I had a suspicion it was glitching because of background transparency, but that doesn't seem to be it. Another possibility is that one of my experiments had left MediaCopy printing images in HLS instead of RGB, but that doesn't seem to meet the fact that the broken output definitely uses RGB. (Unless... I wonder if Level 2 HLS printing is broken on the VT340. I'll have to test that some time.)

unexpected.png

[j4james]

That's brilliant, thanks. I was a bit worried it had something to do with my choice of color scheme. 😊

But getting back to our earlier discussion of DECGRA, I just remembered there was something I wanted to ask ask you about that. If you can set DECGRA multiple times, and it's the last one that takes effect (at least prior to any sixel data output), what happens when you change the aspect ratio?

For example, consider this sequence:

printf '\e[H\ePq"20;1;60;60-"1;1;240;240#1!10~\e\\\n'

It's starting with a 20:1 aspect ratio and a size of 60x60. That's followed by a DECGNL, which assumedly doesn't trigger the background fill, but should move down 120 pixels/6 lines given the 20:1 AR.

We then have another DECGRA changing the aspect ratio to 1:1, and the size to 240x240, followed by some sixel data. It's at this point that background fill would be triggered (assumedly using the most recent size, so 240x240), but is the sixel data also output with the most recent 1:1 aspect ratio, or the previously used 20:1?

In short, do we get a 10x6 block with a 240x240 background fill that is 6 rows down, or is it a 10x120 block, or something else entirely?

[hackerb9]

A 10x6 block with a 240x240 background fill that is 6 rows down.

[j4james]

That's actually kind of useful I think. It gives you a way to position images on the screen in Sixel Display Mode in a reasonably efficient manner. For example, if you wanted to position an image 18 rows down, you don't have to output 60 DECGNL commands - just set the aspect ratio to 60:1, output one DECGNL, and then set it back to the AR that you really want.

I think it might be worth creating another test that demos some of the edge cases involving multiple DECGRA commands that we've discussed in this issue.

[j4james]

FYI, I think you accidentally deleted unexpected.png in your last commit (30a00fafce543a27e27a500acc615dbbd49188b8).