Colors doesn't look right!

[PanosTrak]

Colors doesnt look right, thats the theme i used https://raw.githubusercontent.com/Binaryify/OneDark-Pro/master/themes/OneDark-Pro.json.

[asottile]

hmmm we saw a similar thing when putty was being used -- it looks like some color is being used but the escape sequences are incorrect

can you echo $TERM and $COLORTERM and also try babi-textmate-demo babi/_types.py? my guess is that curses is buggy

[PanosTrak]

[asottile]

can you try this script:

import curses

def c_main(stdscr):
    curses.use_default_colors()

    stdscr.addstr(0, 0, f'can_change_color: {curses.can_change_color()}')
    stdscr.addstr(1, 0, f'colors: {curses.COLORS}')

    stdscr.addstr(3, 0, f'256 color test (should be a blue)')
    curses.init_pair(1, -1, 27)
    stdscr.addstr(4, 0, ' ' * 20, curses.color_pair(1))

    stdscr.addstr(6, 0, 'true color test (should be a slightly different blue)')
    curses.init_color(
        255,
        int(0x1e * 1000 / 255),
        int(0x77 * 1000 / 255),
        int(0xd3 * 1000 / 255),
    )
    curses.init_pair(2, -1, 255)
    stdscr.addstr(7, 0, ' ' * 20, curses.color_pair(2))

    stdscr.get_wch()

if __name__ == '__main__':
    exit(curses.wrapper(c_main))

it should look something like this:

[PanosTrak]

[asottile]

interesting, the terminal reports it can change the color, but it can't!

the colors before actually make a lot more sense now, those are the default 256-color colors, (babi chooses ones at the end in the greyscale zone to change) and since it can't change them you get a weird greyscale "theme"

I wonder if there's a way to detect this and fall back to the 256color rendering 🤔

[asottile]

I have a terrible idea

import curses
import os
import sys
import tempfile

def c_main(stdscr):
    curses.use_default_colors()
    curses.init_color(
        255,
        0x1e * 1000 // 0xff,
        0x77 * 1000 // 0xff,
        0xd3 * 1000 // 0xff,
    )
    curses.init_pair(1, 255, -1)
    stdscr.insstr(0, 0, 'hello world', curses.color_pair(1))
    stdscr.get_wch()

def main():
    saved = os.dup(sys.stdout.fileno())
    with tempfile.TemporaryFile(buffering=False) as tmp:
        os.dup2(tmp.fileno(), sys.stdout.fileno())
        try:
            curses.wrapper(c_main)
        finally:
            os.dup2(saved, sys.stdout.fileno())
        print(tmp.tell())
        tmp.seek(0)
        print(tmp.read())

if __name__ == '__main__':
    exit(main())

this is the start of it (a proof of concept) and it doesn't work yet

the idea is to:

1. save stdout file descriptor
2. swap the stdout file descriptor with a temporary file
3. filter the output, replacing the "change color" sequences and then replacing the color they are replacing with a true color escape sequence (haven't done this yet, I think it would need to happen in a background thread)

[asottile]

this is actually very very close to working:

import contextlib
import curses
import functools
import os
import re
import sys
import tempfile
import threading
from typing import Generator
from typing import Match

CHANGE_COLOR_RE = re.compile(
    br'\033]4;(?P<color>\d+);rgb:'
    br'(?P<r>[0-9A-Fa-f]+)/(?P<g>[0-9A-Fa-f]+)/(?P<b>[0-9A-Fa-f]+)'
    br'\033\\'
)
ESC_256_RE = re.compile(br'\033\[(?P<fgbg>[34]8);5;(?P<color>\d+)m')

def gen(fd: int) -> Generator[bytes, None, None]:
    bts = os.read(fd, 1024)
    while bts:
        yield bts
        bts = os.read(fd, 1024)

def brrr(saved: int, fd: int) -> None:
    colors: Dict[bytes, bytes] = {}

    def sub_color(match: Match[bytes]) -> bytes:
        color = colors.get(match['color'])
        if color is None:
            return match[0]
        else:
            return b'\033[' + match['fgbg'] + b';2;' + color + b'm'

    for chunk in gen(fd):
        for match in CHANGE_COLOR_RE.finditer(chunk):
            r = int(match['r'], 16)
            g = int(match['g'], 16)
            b = int(match['b'], 16)
            colors[match['color']] = f'{r};{g};{b}'.encode()

        chunk = ESC_256_RE.sub(sub_color, chunk)
        os.write(saved, chunk)

def c_main(stdscr):
    if curses.has_colors():
        curses.use_default_colors()
        if curses.can_change_color():
            curses.init_color(
                255,
                0x1e * 1000 // 0xff,
                0x77 * 1000 // 0xff,
                0xd3 * 1000 // 0xff,
            )
            curses.init_pair(1, 255, -1)
    stdscr.insstr(0, 0, 'hello world', curses.color_pair(1))
    stdscr.get_wch()

@contextlib.contextmanager
def fixup_true_color_escapes() -> Generator[None, None, None]:
    saved = os.dup(sys.stdout.fileno())
    r, w = os.pipe()
    thread = threading.Thread(target=functools.partial(brrr, saved, r))

    thread.start()
    os.dup2(w, sys.stdout.fileno())
    try:
        yield
    finally:
        os.dup2(saved, sys.stdout.fileno())
        os.close(w)
        thread.join()

def main():
    with fixup_true_color_escapes():
        curses.wrapper(c_main)

if __name__ == '__main__':
    exit(main())

still need to fix partial sequences aren't handled properly (an escape sequence split across multiple os.read(...) calls)

but here's babi in true color mode running in Konsole (notice the white background on some things, that's the partial sequences problem)

there's also something weird about resizing not working as expected that I'll have to look into 🤔

[ClasherKasten]

I don't know if already known, but when I set $TERM to screen-256color and $COLORTERM to truecolor, everything works as expected. but the test script above doesnt working anymore and gives the following error message:

Traceback (most recent call last):
  File "/home/clasherkasten/test.py", line 27, in <module>
    exit(curses.wrapper(c_main))
  File "/usr/lib/python3.10/curses/__init__.py", line 94, in wrapper
    return func(stdscr, *args, **kwds)
  File "/home/clasherkasten/test.py", line 14, in c_main
    curses.init_color(
_curses.error: init_extended_color() returned ERR

Question: Can this be a practical solution? (As you see on the right a gnome-terminal with xterm-256color and on the right Konsole with screen-256color)

[asottile]

screen-256color (even with the truecolor set) falls back to 256 only color (which is why it appears to work since it doesn't do color reassignment) -- this'll make the colors slightly off from the 24bit colors but they'll at least not be greys