sixel support for images with pixel resolution

srlehn commented 5 years ago

addresses #213

[x] sixel support implemented
[x] check for sixel support (only ANSI terminals considered) otherwise use fallback
[x] update Render()
[x] handle partially shown images
[x] get correct dimensions from xterm
[x] fix: go-tty needed PRs: ~~#15~~, ~~#16~~, ~~#21~~; open, no fix yet: #20
[ ] test image widget with dynamic sizes
[ ] evaluate if external dependencies are necessary or if they could be replaced through some code
[ ] support terminal multiplexer & split view manager tmux, screen, ~~abduco~~ & dvtm, mtm, neercs, twin
[ ] improve rendering: don't "stack" up rendering calls if the drawing is too slow
[ ] improve rendering: draw order after windows resize is sometimes messed up and ends in an empty image widget, clicking on the UI resolves this.
[ ] fix queryTerm() function - often the terminal bugs out and needs a "stty echo". I wasn't yet able to figure it out. If someone could fix this crash demo - that would help a lot.
[ ] split image in multiple subimages if necessary in drawSixel()
[ ] test with changing image sizes and multiple images
[ ] test more terminal emulators: ~~mlterm~~, ~~xterm~~, tmux, screen, mintty, conhost, yaft, iTerm2, macterm
[ ] handle color support for sixel

additional drawers:

[x] iTerm2, MacTerm (untested)
[x] Kitty (remote transmission)
[x] urxvt pixbuf
[ ] conhost via GDI
[ ] X11 like w3imgdisplay perhaps with xgb(util)? (possibly remote?)

This needs the current https://github.com/mattn/go-tty!

Terminals with sixel support

mlterm, mintty (cygwin terminal), xterm with configuration, yaft

https://github.com/saitoha/libsixel/#terminal-requirements

demo

demo_snake

srlehn commented 5 years ago

Mintty panic (go-tty #20)

mintty_panic

problem is somewhere in queryTerm() or tty_windows.go (go-tty)

I think that handling events both with termbox and go-tty is bad but I couldn't make it work in a different way. Probably someone with more Go +/ signals experience finds the answer.

srlehn commented 5 years ago

the important ANSI escape strings here are:

"\033[0c" for querying the terminal capabilities - we need a 4 for sixel.

We also need to know what the size of a character box is in pixels this differs from terminal to terminal:

"\033[14t" gives us the terminal size in pixels

"\033[18t" gives us the terminal size in cells

from that we can calculate the cell size in pixels.

We set the position with "\033[%d;%dH" before printing the sixel string. the position string is simply prepended.

The first %d is the Y position (in cells) and the second the X position.

srlehn commented 5 years ago

xterm works now. The dimensions are now first queried via TIOCGWINSZ (unix only) and then with \033[14t and \033[18t.

current state:

demo

the rendering with images is slower and seems to "stack up"(?) - we should somehow cancel rendering (only from resize or general?) when there is already a newer event.

srlehn commented 5 years ago

opened "screen" pass through issue #56063

==> chromium-hterm forum

==> ocs52.sh

# Send a DCS sequence through screen.
# Usage: <sequence>
screen_dcs() {
  # Screen limits the length of string sequences, so we have to break it up.
  # Going by the screen history:
  #   (v4.2.1) Apr 2014 - today: 768 bytes
  #   Aug 2008 - Apr 2014 (v4.2.0): 512 bytes
  #   ??? - Aug 2008 (v4.0.3): 256 bytes
  # Since v4.2.0 is only ~4 years old, we'll use the 256 limit.
  # We can probably switch to the 768 limit in 2022.
  local limit=256
  # We go 4 bytes under the limit because we're going to insert two bytes
  # before (\eP) and 2 bytes after (\e\) each string.
  echo "$1" | \
    sed -E "s:.{$(( limit - 4 ))}:&\n:g" | \
    sed -E -e 's:^:\x1bP:' -e 's:$:\x1b\\:' | \
    tr -d '\n'
}

==> todo: make screen sixel size dependent on screen version

srlehn commented 5 years ago

initial tmux support - still buggy:

pass through of escape codes through tmux to the terminal works but I assume that the terminal response doesn't get back through tmux (?)

pass through works by wrapping: "\033Ptmux;" + strings.Replace(str, "\033", "\033\033", -1) + "\033\\"

changed draw order: ASCII art first in case sixel fails

srlehn commented 5 years ago

conhost issue: https://github.com/Microsoft/console/issues/120

srlehn commented 5 years ago

iTerm2 sixel support: https://github.com/gnachman/iTerm2/commit/0d0003d9d20b7de3a3ffd815950acb7cc9777c38

srlehn commented 5 years ago

screen can already pass escape sequences <= ~770 bytes through to the terminal. the string has to be prepended by '\033P' and appended by '\033\'.

srlehn commented 5 years ago

Displaying an image through screen!

sed 's,.\{767\},&^[\\^[P,g;1s/^/^[P/;$s/$/^[\\/' snake.sixel

the ^[ are ESC chars (\033 or \x1b) 767 bytes/chars seems to be the max, perhaps it is lower.

srlehn commented 5 years ago

tmux - pixel related: https://github.com/tmux/tmux/issues/1391

srlehn commented 5 years ago

yaft

https://github.com/uobikiemukot/yaft/issues/43

https://github.com/uobikiemukot/yaft/blob/21b69124a2907ad6ede8f45ca96c390615e3dc0c/conf.h#L26

todo: add TERM == "yaft-256color" check for sixelCapable bool

srlehn commented 5 years ago

notes for other terminals

check for TERM == "xterm-kitty"

macterm

sixel & iTerm2 graphics

check TERM_PROGRAM == "MacTerm" (TERM_PROGRAM_VERSION is the build date YYYYMMDD)

srlehn commented 5 years ago

mintty sixel scrolling has changed: https://github.com/mintty/mintty/commit/914e0a1e306a06075bf90a68d92a9b3a1ee5d5e3 (https://github.com/mintty/mintty/issues/866)

srlehn commented 5 years ago

RLogin (Windows) homepage github sixel, regis

srlehn commented 5 years ago

urxvt with sixel patch or from the fork (older)

How to build:

cvs -z3 -d :pserver:anonymous@cvs.schmorp.de/schmorpforge co rxvt-unicode
cd rxvt-unicode/
wget https://gist.github.com/saitoha/be56f3b58c5212abe3f76a13578a548d/raw/01f7c9df2bc8d971f2b3227af89bd95419f25add/rxvt-unicode-sixel.patch
patch -p1 <rxvt-unicode-sixel.patch
sed -i '/^#include <stdio.h>/a #include <sys/stat.h>' src/screen.C
./autogen.sh
./configure --prefix=/usr/
make

srlehn commented 5 years ago

ranger img_display.py Wiki Image Previews implementations for w3mimgdisplay, urxvt pixbuf, kitty, Terminology

man 7 urxvt for escape sequences

srlehn commented 5 years ago

added support for urxvt (pixbuf backgrounds), kitty (remote transmission) and MacTerm

srlehn commented 5 years ago

notes for xterm sixel implementation:

https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Functions-using-CSI-_-ordered-by-the-final-character_s_

CSI ? Pi ; Pa ; Pv S
          If configured to support either Sixel Graphics or ReGIS Graph-
          ics, xterm accepts a three-parameter control sequence, where
          Pi, Pa and Pv are the item, action and value:

            Pi = 1  -> item is number of color registers.
            Pi = 2  -> item is Sixel graphics geometry (in pixels).
            Pi = 3  -> item is ReGIS graphics geometry (in pixels).

            Pa = 1  -> read
            Pa = 2  -> reset to default
            Pa = 3  -> set to value in Pv
            Pa = 4  -> read the maximum allowed value

            Pv can be omitted except when setting (Pa == 3 ).
            Pv = n <- A single integer is used for color registers.
            Pv = width ; height <- Two integers for graphics geometry.

          xterm replies with a control sequence of the same form:

               CSI ? Pi ; Ps ; Pv S

          where Ps is the status:
            Ps = 0  -> success.
            Ps = 1  -> error in Pi.
            Ps = 2  -> error in Pa.
            Ps = 3  -> failure.

          On success, Pv represents the value read or set.

          Notes:
          o   The current implementation allows reading the graphics
              sizes, but disallows modifying those sizes because that is
              done once, using resource-values.
          o   Graphics geometry is not necessarily the same as "window
              size" (see the dtterm window manipulation extensions).
              For example, xterm limits the maximum graphics geometry at
              compile time (1000x1000 as of version 328) although the
              window size can be larger.
          o   While resizing a window will always change the current
              graphics geometry, the reverse is not true.  Setting
              graphics geometry does not affect the window size.

==> TODO: Break image into subimages not larger than 1000x1000 for xterm. Size should be a multiple of cell size in pixels dividable by six this is because we put the cursor position by cell not by pixel and want gaps as small as possible if any.

srlehn commented 5 years ago

images in conhost:

idea from http://www.cplusplus.com/forum/beginner/223667/#msg1024626 (by JLBorges)

// http://www.cplusplus.com/forum/beginner/223667/#msg1024626
// x86_64-w64-mingw32-g++-win32 conhostpixel.cc -lws2_32 -lgdi32 -user32 -static-libstdc++ -static-libgcc

#include <iostream>
#include <cstdlib>
#include <stdio.h>
#include <windows.h>

int main() {
    const auto console = ::GetConsoleWindow() ;
    const auto context = ::GetDC(console) ;
    constexpr auto red = RGB( 255, 0, 0 ) ;
    constexpr auto yellow = RGB( 255, 255, 0 ) ;

    for( int i = 2 ; i < 200 ; ++i ) for( int j = 2 ; j < 200 ; ++j )
        ::SetPixel( context, i, j, std::abs(i-j) > 25 ? red : yellow );

    Sleep(5000);
}

Go library: https://github.com/mattn/drawcmd (image property of mattn) screenshot

diamondburned commented 5 years ago

I'm working on a go-w3m library here: https://gitlab.com/diamondburned/go-w3m/ Along with a port of Ueberzug in Golang (that I'm working on)

srlehn commented 5 years ago

I'm working on a go-w3m library here: https://gitlab.com/diamondburned/go-w3m/ Along with a port of Ueberzug in Golang (that I'm working on)

Both are interesting libraries, thanks for creating them. I already planned implementing a drawer with xgb myself (was too much effort for me atm though learning X11...). Would it be possible to license the w3m code under the MIT like termui? I do like copyleft-licenses like the MPL but I don't think it is wished for to include copyleft in termui. The ueberzug-go library is currently without license so not usable please add a license (if possible MIT/BSD). The panics in the ueberzug-go should in my opinion be replaced by errors. I think an application should do the decision to panic not a library.

diamondburned commented 5 years ago

Latest commit changed go-w3m's LICENSE to MIT. I'm still working on ueberzug's resize feature, which sadly doesn't work at the moment.

seebye commented 5 years ago

@diamondburned I have a few problems with ueberzug-go.

I'm not sure whether it's also right for the GPLv3, but at least as far as I know it's right for the GPLv2.
If you port a program to another language you have also to license it with the GPL otherwise you're violating the GPL.
e.g. see: https://softwareengineering.stackexchange.com/a/151516
You're pretending to port ueberzug to golang (e.g. by your chosen name ueberzug-go),
but your code suggest otherwise.
The only communality is that you're also using child windows to display images.

diamondburned commented 5 years ago

Already stated that ueberzug-go is a work in progress, and I don't plan on mimicking any of ueberzug's features. All it is, is something that displays images the way ueberzug does it, but in Go.

seebye commented 5 years ago

Along with a port of Ueberzug in Golang

(1) Well, that didn't sound like that.
If it isn't a port you shouldn't imply it to be one by nameing it "ueberzug-go".
If it is one you should use the GPL.
That's all.

All it is, is something that displays images the way ueberzug does it, but in Go.

(2) Depends on how similar you want to do that:
Every file is licensed with the GPL. So if you just port some of the code the GPL applies.
Your code currently however doesn't look like a port -> (1).

Already stated that ueberzug-go is a work in progress

I know. That's why I mentioned (1) and (2).
To sum it up:

In case of porting code: It's not okay to violate the GPL
In case of not being a port: It's not okay to pretend a project to be something it isn't -> you should choose another name

Edit:

I'm not pretending to be a port. I'll keep this as it is.

...

Along with a port of Ueberzug in Golang https://github.com/gizak/termui/pull/233#issuecomment-487452344

A simple half-assed port of ueberzug to Golang [...] https://gitlab.com/diamondburned/ueberzug-go/blob/master/README.md#L3

No, you clearly aren't pretending it.. However as the project isn't worth more effort than telling people not to behave like a dick, I will leave it at this point to write further comments on this topic.

diamondburned commented 5 years ago

I'm not pretending to be a port. I'll keep this as it is.

ghost commented 4 years ago

Is sixel support now working with termui? If so, would be the only system I know besides Jexer to properly mix text and graphics. It would be great to have more toolkits doing this.

I have found several more terminals with sixel support: https://gitlab.com/klamonte/jexer/wikis/terminals

Also, vte and xterm.js are testing sixel support now, and alacritty is implementing. It is likely in the 3-12 month time frame that vastly more terminals will be out there that support sixel.

diamondburned commented 3 years ago

Just confirmed this works on foot.

ghost commented 3 years ago

Couple FYIs:

Another TUI system with sixel output is notcurses: https://github.com/dankamongmen/notcurses/issues/1380
alacritty's sixel test branch is working great.
contour and foot are two more terminals with sixel output working great.
vte's sixel is working and in their git head. No idea when it makes it to the distros.

diamondburned commented 3 years ago

I'm currently working on a tcell pull request and a SIXEL library for it:

https://github.com/diamondburned/tcell-sixel https://github.com/gdamore/tcell/pull/436

timsofteng commented 3 years ago

Any news about it? Will we see sixel or kitty image protocol in tmux?

ghost commented 2 years ago

@srlehn @diamondburned

This looks fantastic, awesome job! :-)

I wanted to pass on some things I've picked up recently, on the odd chance it could help. (Even if just by way of "go check out notcurses, it's really cool." ;) )

I added translucent windows with images under/over, thanks in large part to some clues from notcurses. Some notes of that work are here: https://gitlab.com/klamonte/jexer/-/issues/88 . I also wrote down more details on my particular mixed image-and-text cell model here , with some narrative (and again the notcurses shoutouts) here.

For encoding to sixel, check out notcurses' new octree implementation sometime. It's very likely the fastest and highest-quality anywhere right now, at least for xterm-type terminals. (libsixel was tested on real hardware, so that might give it an edge still in terms of compatibility.)

Depending on how you choose to render, you may find that 256-bit colors per sixel could still result in close to 16-bit apparent visual bit depth on the actual screen, making the drive for non-sixel support a bit less crucial. (Note that the best bit depth for sixel is 19.97 bits: 101^3.) The screenshot link is my very naive median-cut encoder, which is much worse than notcurses' octree encoder -- but that's still good enough that you might not be able to tell when I'm using iTerm2+PNG vs sixel.

Finally, and to my utter surprise, I may have encountered an actual xterm bug: https://gitlab.com/klamonte/jexer/-/issues/89 . Both foot and wezterm are doing quite well on sixel, and very nice options to have for testing.

srlehn commented 1 year ago

This PR transformed into this library: https://github.com/srlehn/termimg

A widget for termui can be found in the tui/termuiimg subfolder.

gizak / termui