Add some manner of Sixel interface

[dankamongmen]

I'm not yet deeply knowledgeable regarding Sixel, but it seems pretty tight (where supported). It's definitely not a 1.0.0 thing, but we ought open up the Sixel floodgates sooner rather than later IMHO.

[dankamongmen]

libsixel looks pretty solid, and is MIT-licensed: https://github.com/dankamongmen/libsixel

so sixel seems sweet. Problem is lack of support -- it's not present in kitty, alacritty, or vte, at least. To get it working well in xterm, you need:

XTerm*decTerminalID: vt340
XTerm*numColorRegisters: 256

in your xresources database, or to launch it with:

xterm -ti vt340

(we can upgrade the number of color registers ourselves, ala lsix, so long as vt340 mode is used)

but yeah, that's just xterm. it does work over ssh, though.

[dankamongmen]

And now, of course, it is obvious how we ought integrate Sixel: as a glyph blitting backend of ncvisual (see #622). Huzzah!

[dankamongmen]

it looks like we'll want to use sixel_encode()

/* convert pixels into sixel format and write it to output context */
SIXELAPI SIXELSTATUS
sixel_encode(
    unsigned char  /* in */ *pixels,     /* pixel bytes */
    int            /* in */  width,      /* image width */
    int            /* in */  height,     /* image height */
    int            /* in */  depth,      /* color depth: now unused */
    sixel_dither_t /* in */ *dither,     /* dither context */
    sixel_output_t /* in */ *context);   /* output context */

we'll need to provide an output context:

/* create output context object */
SIXELAPI SIXELSTATUS
sixel_output_new(
    sixel_output_t          /* out */ **output,     /* output object to be created */
    sixel_write_function    /* in */  fn_write,     /* callback for output sixel */
    void                    /* in */ *priv,         /* private data given as
                                                       3rd argument of fn_write */
    sixel_allocator_t       /* in */  *allocator);  /* allocator, null if you use
                                                       default allocator */

[dankamongmen]

I spent some time looking through libsixel, and am unimpressed. It's lousy with scan-build errors and indeed even some basic compiler warnings. It also has a ton of crap we don't need. I'm thinking we just hand-write the NCBLIT_SIXEL blitter.

And we still don't seem to have a way to detect whether Sixel is available :/.

[dankamongmen]

lsix appears to be capable of determining this:

Error: Your terminal does not report having sixel graphics support.

Please use a sixel capable terminal, such as xterm -ti vt340, or
ask your terminal manufacturer to add sixel support.

You may test your terminal by viewing a single image, like so:

        convert  foo.jpg  -geometry 800x480  sixel:-

If your terminal actually does support sixel, please file a bug
report at http://github.com/hackerb9/lsix/issues

(Please mention device attribute codes: ^[[?6c)

[dankamongmen]

JFC, lsix is a bash script? heavens save us.

    # IS TERMINAL SIXEL CAPABLE?    # Send Device Attributes
    IFS=";" read -a REPLY -s -t 1 -d "c" -p $'\e[c' >&2
    for code in "${REPLY[@]}"; do
  if [[ $code == "4" ]]; then
      hassixel=yup
      break
  fi
    done

ugh, it looks like it requires interrogating the terminal :/

[dankamongmen]

Yeah, drop the libsixel crap. We'll proceed directly.

[dankamongmen]

The only terminals I can find with support for this are Xterm and mlterm (though maybe the OSX and Windows terminals have support; I'm unsure). Hrmm.

[dankamongmen]

We are now detecting Sixel support dynamically. If NCOPTION_DETECT_SIXEL is provided to notcurses_init(), we detect it upon startup. Otherwise, we detect it the first time (and only the first time) we attempt to use NCBLIT_SIXEL. If NCBLIT_SIXEL is provided together with NCVISUAL_OPTIONS_NODEGRADE, and Sixel is unavailable, the render attempt will fail.

[dankamongmen]

Just stumbled across OCS1337, supported by hterm and iterm

[dankamongmen]

OK, I've got all the detection and such working in dankamongmen/shitxel. I just don't see this being useful enough to finish out and commit at the moment. Perhaps we'll come back to it.

[dankamongmen]

Sixel Graphics If xterm is configured as VT240, VT241, VT330, VT340 or VT382 using the decTerminalID resource, it supports Sixel Graphics controls, a palleted bitmap graphics system using sets of six vertical pixels as the basic element.

CSI Ps c Send Device Attributes (Primary DA), xterm. xterm responds to Send Device Attributes (Primary DA) with these additional codes: Ps = 4 ⇒ Sixel graphics.

CSI ? Pm h Set Mode, xterm. xterm has these additional private Set Mode values: Ps = 8 0 ⇒ Sixel scrolling. Ps = 1 0 7 0 ⇒ use private color registers for each graphic. Ps = 8 4 5 2 ⇒ Sixel scrolling leaves cursor to right of graphic.

DCS Pa ; Pb ; Ph q Ps..Ps ST Send SIXEL image, DEC graphics terminals, xterm. See:

         VT330/VT340 Programmer Reference Manual Volume 2:
         Graphics Programming
         Chapter 14 Graphics Programming

      The sixel data device control string has three positional
      parameters, following the q  with sixel data.
        Pa ⇒  pixel aspect ratio
        Pb ⇒  background color option
        Ph ⇒  horizontal grid size (ignored).
        Ps ⇒  sixel data

[dankamongmen]

CSI
? P i ; P a ; P v S
Set or request graphics attribute, xterm. If configured to support either Sixel Graphics or ReGIS
Graphics, xterm accepts a three-parameter control sequence, where P i , P a and P v are the item,
action and value:
Patch #350
8
2019/11/02XTerm Control Sequences
VT100 Mode
P i = 1 → item is number of color registers.
P i = 2 → item is Sixel graphics geometry (in pixels).
P i = 3 → item is ReGIS graphics geometry (in pixels).
P a = 1 → read attribute.
P a = 2 → reset to default.
P a = 3 → set to value in P v .
P a = 4 → read the maximum allowed value.
P v can be omitted except when setting (P a == 3 ).
P v = n ← A single integer is used for color registers.
P v = width ; height ← Tw o integers for graphics geometry.
xterm replies with a control sequence of the same form:
CSI
? P i ; P s ; P v S
where P s is the status:
P s = 0 ← success.
P s = 1 ← error in P i .
P s = 2 ← error in P a .
P s = 3 ← failure.
On success, P v represents the value read or set.
Notes:
• The current implementation allows reading the graphics sizes, but disallows modifying those
sizes because that is done once, using resource-values.
• Graphics geometry is not necessarily the same as “window size” (see the dtterm window manip-
ulation extensions). For example, xterm limits the maximum graphics geometry at compile time
(1000x1000 as of version 328) although the window size can be larger.
• While resizing a window will always change the current graphics geometry, the reverse is not
true. Setting graphics geometry does not affect the window size.

[joseluis]

I just tried running notcurses-demo on xterm -ti vt340 (or just xterm really) and the demos look very bad for me. Do they work well for you?

[dankamongmen]

I just tried running notcurses-demo on xterm -ti vt340 (or just xterm really) and the demos look very bad for me. Do they work well for you?

They definitely don't look great, but I'm accustomed to xterm being the hardest terminal to get things "looking right" in. -ti vt340 doesn't look any worse than normal xterm....oh, i'm using vt340 by default. Here's my .Xresources:

[schwarzgerat](0) $ cat .Xresources 
URxvt*termName: rxvt-256color
URxvt*scrollBar: false
URxvt*scrollBar_right: false
URxvt*scrollBar_floating: false
URxvt*background: Black
URxvt*foreground: White

xterm*background: black
xterm*foreground: lightgray
xterm*bidi.enabled: 0
!XTerm.vt100.faceNameDoublesize: WenQuanYi WenQuanYi Bitmap Song
!XTerm.vt100.faceSize: 12
!XTerm*vt100.renderFont: false
!xterm*font: *-fixed-*-*-*-18-*
xterm*directColor: true
XTerm*decTerminalID: vt340
XTerm*numColorRegisters: 256
XTerm*utf8: 1
XTerm*eightBitInput: false
XTerm*allowWindowOps: False
XTerm*disallowedWindowOps: 1,2,3,4,5,6,7,8,9,11,13,18,19,20,21,GetSelection,SetSelection,SetWinLines,SetXprop
[schwarzgerat](0) $ 

what badness are you seeing? screenshot?

[joseluis]

what badness are you seeing? screenshot?

P.S. That's using your .Xresources, it makes no difference for me.

[dankamongmen]

what badness are you seeing? screenshot?

P.S. That's using your .Xresources, it makes no difference for me.

no, lol, nothing looks this bad for me on xterm. ugh.

btw, with that xresources, you've got to either restart x or run xrdb -a $HOME/.Xresources for it to take effect. xterm is lovely.

[dankamongmen]

It looks like it's possible to detect Sixel support on a terminal which implements "Send Device Attributes" at the vt200 level or higher:

CSI Ps c  Send Device Attributes (Primary DA).
            Ps = 0  or omitted ⇒  request attributes from terminal.  The
          response depends on the decTerminalID resource setting.
            ⇒  CSI ? 1 ; 2 c  ("VT100 with Advanced Video Option")
            ⇒  CSI ? 1 ; 0 c  ("VT101 with No Options")
            ⇒  CSI ? 4 ; 6 c  ("VT132 with Advanced Video and Graphics")
            ⇒  CSI ? 6 c  ("VT102")
            ⇒  CSI ? 7 c  ("VT131")
            ⇒  CSI ? 1 2 ; Ps c  ("VT125")
            ⇒  CSI ? 6 2 ; Ps c  ("VT220")
            ⇒  CSI ? 6 3 ; Ps c  ("VT320")
            ⇒  CSI ? 6 4 ; Ps c  ("VT420")

          The VT100-style response parameters do not mean anything by
          themselves.  VT220 (and higher) parameters do, telling the
          host what features the terminal supports:
            Ps = 1  ⇒  132-columns.
            Ps = 2  ⇒  Printer.
            Ps = 3  ⇒  ReGIS graphics.
            Ps = 4  ⇒  Sixel graphics.
            Ps = 6  ⇒  Selective erase.
            Ps = 8  ⇒  User-defined keys.
            Ps = 9  ⇒  National Replacement Character sets.
            Ps = 1 5  ⇒  Technical characters.
            Ps = 1 6  ⇒  Locator port.
            Ps = 1 7  ⇒  Terminal state interrogation.
            Ps = 1 8  ⇒  User windows.
            Ps = 2 1  ⇒  Horizontal scrolling.
            Ps = 2 2  ⇒  ANSI color, e.g., VT525.
            Ps = 2 8  ⇒  Rectangular editing.
            Ps = 2 9  ⇒  ANSI text locator (i.e., DEC Locator mode).

[dankamongmen]

See also #1095 regarding Kitty's bespoke pixel-based solution. Now that we're doing dreadful per-TERM heuristics, I think there's more of a place for this (also, given @grendello 's OSX work, we'll soon be targeting ITerm, which supports Sixel IIRC). Marking this due for 2.3.0.

[joseluis]

Good news, there's already a PR for sixel support in Alacritty.

In order to run it:

git clone https://github.com/ayosec/alacritty
git checkout graphics
cargo run --release

• test1: boxes.sh
• test2: video player in ruby

[dankamongmen]

Good news, there's already a PR for sixel support in Alacritty.

In order to run it:

git clone https://github.com/ayosec/alacritty
git checkout graphics
cargo run --release

• test1: boxes.sh
• test2: video player in ruby

awesome, that definitely ups the priority for this. if both alacritty and kitty have a pixel-graphics implementation, i can see tools starting to roll out for them. thanks for the heads up @joseluis !

[dankamongmen]

hey @joseluis , do things still look that FUBAR on xterm for you? surprised i never chased that down. i think it ought be started up as another bug if you're still seeing it.

on to sixel!

[dankamongmen]

in the dankamongmen/sixel-redux branch, we're now detecing sixel support with Report Device Attributes, but only if NCOPTIONS_VERIFY_SIXEL is explicitly passed to notcurses_init(). it's not yet detected in direct mode.

[dankamongmen]

Just need to implement sixel_blit() now.

[dankamongmen]

so everyone else has got a much simpler problem: lsix etc just throw something up on the screen directly, ala our direct mode. we of course are composing planes, planes backed by nccell virtual framebuffers. how do we intend to code sixel into nccells?

my first thought is that we'll use a new non-printing sentinel, presumably 2. 2 will map into the egcpool just like 1 does, but indicates a sixel (or maybe 2? need figure out how exactly sixels map to display cells). there are then two complications:

(1) when rendering, how do we integrate a sixel with a CELL_ALPHA_BLEND above us? how do we integrate a CELL_ALPHA_BLEND sixel with a cell below it? however we do, it will lead to some nccell in the rendered frame... (2) when rasterizing a frame, how do we integrate sixels? presumably we enter the sixel state, meaning we'll have to exit said state when we get back to non-sixels. while in this state, we don't emit colors. how do we take advantage of sixel RLE?

[joseluis]

hey @joseluis , do things still look that FUBAR on xterm for you? surprised i never chased that down. i think it ought be started up as another bug if you're still seeing it.

I've just tried it notcurses-demo looks very good on xterm now (except for the lack of truecolor, and of many unicode symbols that are shown as boxes, but that's probably my setup's fault).

[dankamongmen]

iirc, xterm*directColor: true is necessary for truecolor on xterm. thanks for the recheck!

[dankamongmen]

OK, I've amended the rasterizer to enter and leave sixel mode based off of cell_pixels_p(). This latter is based off a new bit test, CELL_PIXEL_GRAPHICS, 0x0000000080000000. this bit can be set with cell_set_pixels(), which will be set by the sixel blitter, which I must now write. This blitter will write some number of sixels to each EGC, which it will then mark with cell_set_pixels(). See #1368 which covers getting the number of sixels per cell.

I'm thinking we maybe always want to perform a cup following leaving pixel mode, since we really don't know exactly where we'll be.

[dankamongmen]

Actually, hrmm, maybe we can exploit this to optimize the color registers. We don't want to have to reload them following each sixel, but grouping them sounds annoying, too (and I don't want to add lookahead to the rasterizer). So maybe we just keep a common set of registers in a cell? Except of course when we run up against the end of a framebuffer row...

[dankamongmen]

Time to go to bed, but we're now generating some manner of pixel output through direct mode. We're outputting a bunch of sixel codes in rendered mode, but that ought be simple enough to resolve. Need to break down the colors; we're currently always providing 100;100;100, but it shouldn't be difficult to break down. We're close!

[dankamongmen]

I merged the first big bit of work -- all the infrastructure and documentation -- just now as #1372. Need to get sixel_blit() working properly now. Let's use lsix output and differential analysis to figure out what's going on there. lsix, btw, seems to happily use a pretty large number of color registers, certainly more than the 4 of the vt340, heh.

[dankamongmen]

getting closer...we can now draw a 20x20 monochromatic square, except that there are gaps between each sixel band, lol. but getting there! and we draw it in both direct and rendered mode.