sc-im seems to change the color of the underlying terminal even after it is closed

[poetaman]

sc-im seems to change the color of underlying terminal even after it is closed. This does not happen with any other application I use with the current setup. I am using the Dracula color scheme. Here's my entire scimrc, perhaps the problem is with REDEFINE_COLOR, though idk.

set autocalc
set numeric
set numeric_zero
set numeric_decimal
set newline_action=j
set truncate
set xlsx_readformulas

# Dracula theme
REDEFINE_COLOR "WHITE" 248 248 242
REDEFINE_COLOR "BLACK" 34 33 34
REDEFINE_COLOR "RED" 255 149 128
REDEFINE_COLOR "GREEN" 138 255 128
REDEFINE_COLOR "YELLOW" 255 255 128
REDEFINE_COLOR "BLUE" 128 255 234
REDEFINE_COLOR "CYAN" 128 255 234
REDEFINE_COLOR "MAGENTA" 255 128 191
DEFINE_COLOR "comment" 121 112 169
DEFINE_COLOR "altbackground" 63 63 63
color "type=DEFAULT fg=WHITE bg=BLACK bold=0 italic=0"
color "type=NORMAL fg=WHITE bg=BLACK bold=0 italic=0"
color "type=WELCOME fg=comment bg=BLACK bold=0"
color "type=MODE fg=GREEN bg=BLACK bold=0"
color "type=GRID_EVEN fg=WHITE bg=BLACK"
color "type=GRID_ODD fg=WHITE bg=BLACK"
color "type=CELL_ID fg=WHITE bg=BLACK bold=0"
color "type=CELL_FORMAT fg=comment bg=BLACK bold=0"
color "type=CELL_CONTENT fg=WHITE bg=BLACK bold=0"
color "type=INFO_MSG fg=GREEN bg=BLACK bold=0"
color "type=ERROR_MSG fg=BLACK bg=RED bold=1"
color "type=INPUT fg=CYAN bg=BLACK"
color "type=CELL_ERROR fg=BLACK bg=RED bold=1"
color "type=HEADINGS fg=comment bg=BLACK bold=0"
color "type=HEADINGS_ODD fg=comment bg=BLACK bold=0"
color "type=CELL_SELECTION fg=WHITE bg=altbackground bold=0"
color "type=CELL_SELECTION_SC fg=WHITE bg=altbackground bold=1"
color "type=NUMB fg=MAGENTA bg=BLACK bold=0"
color "type=CELL_NEGATIVE fg=RED bg=BLACK bold=0"
color "type=EXPRESSION fg=YELLOW bg=BLACK bold=0"
# To not italicize strings, replace italic=1 with italic=0 on following line
color "type=STRG fg=CYAN bg=BLACK bold=0 italic=1"
color "type=DATEF fg=YELLOW bg=BLACK"
color "type=HELP_HIGHLIGHT fg=BLACK bg=WHITE"

# Papercolor dark theme
#REDEFINE_COLOR "WHITE" 208 208 208
#REDEFINE_COLOR "BLACK" 28 28 28
#REDEFINE_COLOR "RED" 255 149 128
#REDEFINE_COLOR "GREEN" 175 215 0
#REDEFINE_COLOR "YELLOW" 255 175 95
#REDEFINE_COLOR "BLUE" 95 175 215
#REDEFINE_COLOR "CYAN" 0 175 175
#REDEFINE_COLOR "MAGENTA" 255 95 175
#DEFINE_COLOR "comment" 0 135 135
#DEFINE_COLOR "altbackground" 88 88 88
#DEFINE_COLOR "infofg" 215 175 95
#DEFINE_COLOR "errorbg" 175 0 0
#
#color "type=DEFAULT fg=WHITE bg=BLACK bold=0 italic=0"
#color "type=NORMAL fg=WHITE bg=BLACK bold=0 italic=0"
#color "type=WELCOME fg=CYAN bg=BLACK bold=0"
#color "type=MODE fg=infofg bg=BLACK bold=0"
#color "type=GRID_EVEN fg=WHITE bg=BLACK"
#color "type=GRID_ODD fg=WHITE bg=BLACK"
#color "type=CELL_ID fg=WHITE bg=BLACK bold=0"
#color "type=CELL_FORMAT fg=comment bg=BLACK bold=0"
#color "type=CELL_CONTENT fg=WHITE bg=BLACK bold=0"
#color "type=INFO_MSG fg=infofg bg=BLACK bold=0"
#color "type=ERROR_MSG fg=WHITE bg=errorbg bold=0"
#color "type=INPUT fg=BLUE bg=BLACK"
#color "type=CELL_ERROR fg=WHITE bg=errorbg bold=0"
#color "type=HEADINGS fg=comment bg=BLACK bold=0"
#color "type=HEADINGS_ODD fg=comment bg=BLACK bold=0"
#color "type=CELL_SELECTION fg=WHITE bg=altbackground bold=1"
#color "type=CELL_SELECTION_SC fg=WHITE bg=altbackground bold=1"
#color "type=NUMB fg=YELLOW bg=BLACK bold=0"
#color "type=CELL_NEGATIVE fg=MAGENTA bg=BLACK bold=0"
#color "type=EXPRESSION fg=GREEN bg=BLACK bold=0"
## To not italicize strings, replace italic=1 with italic=0 on following line
#color "type=STRG fg=BLUE bg=BLACK bold=0 italic=1"
#color "type=DATEF fg=YELLOW bg=BLACK"
#color "type=HELP_HIGHLIGHT fg=BLACK bg=WHITE"

[Gitleptune]

Experimenting with making more themes currently and can confirm that this happens to me as well. It gets fixed when I restart terminal (as expected).

[poetaman]

In an ideal implementation, upon "redefine" sc-im won't change the underlying definition of terminal color. It would rather mark the color as redefined, and save a pointer to the target color. This way even if sc-im crashes, the underlying terminal's colors would remain what they were at the time of opening sc-im.

[andmarti1424]

Hello folks. I am afraid this is an ncurses problem. sc-im do not interact with terminal. It just use ncurses, and let it do the hard stuff. Do that happen also happen to you with other attributes such as bold?

[poetaman]

@andmarti1424 Didn't try with other attributes. Could be that there is some terminfo setting? adding @ThomasDickey for comment, as he is the ncurses maintainer.

[andmarti1424]

No. Not that I can recall.

[andmarti1424]

@reportaman have you contacted @ThomasDickey? I dont think this is a sc-im problem.

[poetaman]

@andmarti1424 There is an answer from @ThomasDickey on stackoverflow from mid-2016. https://stackoverflow.com/a/36053516/15087532

It seems like redefining color palette of terminals is a one way street, unless something changed since mid-2016 it doesn't seem like ncurses can reinstate the terminal settings that were edited by the program that uses it. Perhaps there should not be a REDEFINE_COLOR option to redefine the color palette of terminal (0-15 or 0-255 colors based on 16-bit or 256 terminal mode), or if possible an app should figure out the color it is trying to redefine and then at the time of exit reinstate it. The only caveat of the latter approach is that if the application crashes, the terminal colors will not get reinstated, so former would be a safe bet

I believe new colors that are defined DEFINE_COLOR are not defined using the 16-255 colors of terminals, right? If they are using some of the 16-255 color slots then DEFINE_COLOR too has a problem...

[andmarti1424]

sc-im dont change default terminal background and foreground colors specifically. It just defines a palette of colors and then use them on different parts of screen.

What if you change those REDEFINE_COLOR and defines new colors like you did with altbackground, and comment colors. Does the terminal is left with other colors than default when quitting sc-im?

[poetaman]

@andmarti1424 If I just use DEFINE_COLORs, then it seems terminal dependent: if terminal is iTerm2/WezTerm/AppleTerminal's colors are not affected, but Alacritty's colors are affected just like tmux's in the following statements (and video). The problem inside tmux continues to persist either way. Tmux set's colors using pane-colours array. https://github.com/tmux/tmux/search?q=pane-colours, https://github.com/tmux/tmux/issues/2815. Maybe sc-im is trying to write to the same colors of ncurses (I guess tmux is based on ncurses). Here's a video that shows the problem with sc-im within tmux:

I haven't used any other nurses app to modify colors so can't say if it affects other apps. Am shooting in the dark here, I think the problem will get resolved if 0-255 colors of ncurses are not touched by sc-im...

https://user-images.githubusercontent.com/71736629/134449964-68d973cc-0a8c-4527-b76c-cde074048fe3.mp4

[andmarti1424]

After watching the screencast, the problem you both are having is clear. However, I still think this is not an issue of sc-im, but ncurses. sc-im does not interact with terminal directly, nor does change the terminal background and foreground colors. And I am not talking about if sc-im crashes (if that happens it is known that terminal state is not restored properly), but on clean exits.

ncurses should restore default terminal attributes (not only the fg and bg color) at exit. and that is what endwin() is supposed to do I believe: https://pubs.opengroup.org/onlinepubs/7908799/xcurses/endwin.html

I dont think tmux uses ncurses at all.

[poetaman]

@nicm Please look at the screencast, this problem does not occur in most terminals (accept Alacritty) when sc-im is not called from within tmux, but always happens when sc-im is called from within tmux. It makes me wonder if this is because tmux's pane-colours and sc-im's DEFINE_COLOR both update the same library that defines tmux's colors & if anything can be done about it. @ThomasDickey's inputs will be appreciated too, as @andmarti1424 suspects this has something to do with ncurses.

[nicm]

Show tmux server log please.

[poetaman]

@nicm here we go... tmux-server-4445.log tmux-out-4445.log tmux-client-4443.log

[nicm]

It appears this application is setting the palette with OSC 4:

1632811200.809663 input_exit_osc: "4;8;rgb:F7/F7/F1" (end ST)
1632811200.809704 input_exit_osc: "4;9;rgb:21/20/21" (end ST)
1632811200.809720 input_exit_osc: "4;10;rgb:FE/94/7F" (end ST)
1632811200.809735 input_exit_osc: "4;11;rgb:89/FE/7F" (end ST)
1632811200.809749 input_exit_osc: "4;12;rgb:FE/FE/7F" (end ST)
1632811200.809763 input_exit_osc: "4;13;rgb:7F/FE/E9" (end ST)
1632811200.809777 input_exit_osc: "4;14;rgb:7F/FE/E9" (end ST)
1632811200.809791 input_exit_osc: "4;15;rgb:FE/7F/BE" (end ST)
1632811200.809806 input_exit_osc: "4;16;rgb:78/6F/A8" (end ST)
1632811200.809820 input_exit_osc: "4;17;rgb:3E/3E/3E" (end ST)

But I do not see any corresponding OSC 104 to reset the palette when it exits.

I wondered if the palette should be reset on exiting the alternate screen but from testing in xterm that does not appear to be the case.

In what terminal does it correctly reset the palette on exit?

[poetaman]

@nicm

In what terminal does it correctly reset the palette on exit?

It correctly resets in iTerm2/WezTerm/Apple's Terminal, except on Alacritty where the behavior is the same as in tmux. All tested with their respective latest master branches, on system described below. Didn't try other terminals.

WezTerm & Alacritty are the only cross platform terminals among ones mentioned above,

❯ neofetch
                    'c.          reportaman@macmini.local
                 ,xNMM.          ----------------------------
               .OMMMMo           OS: macOS 11.6 20G165 arm64
               OMMM0,            Host: Macmini9,1
     .;loddo:' loolloddol;.      Kernel: 20.6.0
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Uptime: 2 hours, 31 mins
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    Packages: 1 (brew)
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: zsh 5.8
;MMMMMMMMMMMMMMMMMMMMMMMM:       Resolution: 3840x2160
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Quartz Compositor
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    WM Theme: Blue (Dark)
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal: WezTerm
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   CPU: Apple M1
    kMMMMMMMMMMMMMMMMMMMMMMd     GPU: Apple M1
     ;KMMMMMMMWXXWMMMMMMMk.      Memory: 1980MiB / 16384MiB
       .cooc,.    .,coo:.

[nicm]

OK run the program with script like this please in iTerm2:

script
... reproduce problem ...
exit

Then show me the typescript file.

You may need to install script, can't remember.

[nicm]

That's outside tmux of course.

[poetaman]

@nicm Here's a run from script -r on iTerm2 in macOS, replay it with script -p typescript.txt on macOS or scriptreplay on linux (as per the code in https://github.com/wez/wezterm/blob/main/wt-replay).

typescript.txt

[poetaman]

Actually, for some reason, the scriptreplay doesn't work on linux. Do you have a Mac machine, it works correctly on Mac with script -p typescript.txt?

[nicm]

Can you do it without -r please?

[poetaman]

@nicm macOS script & linux script/ scriptreplay seem very incompatible. If I do not pass -r, it doesn't play correctly on macOS, if I pass -r it plays incorrectly on linux. So I decided to install a few KBs of https://asciinema.org. Its a modern version of script, and in my testing a recording produced on one platform works well on another.

Here's the command to replay file ascii.cast:

>>asciinema play ascii.txt

ascii.txt

The file format is pretty similar to script's format, but seems to work well everywhere...

Note: given iTerm* is not available on linux, its not really testable there. When I run this .cast file in WezTerm, the behavior differs... it doesn't reset the color (on either macOS or Linux (Arch)). Though if I repeat the exact same steps manually, it does reset the colors. The only consistent behavior seems on iTerm2.

[nicm]

I don't care about replaying it, I am going to look at the file, so I don't want any replay crap in it.

[poetaman]

Update: If I run the same script output or asciinema output in iTerm2 on another account in macOS, then it does not reset the palette on exit.

[poetaman]

@nicm Here's a typescript generated using script command without any options, and the video where I show it being recorded & one can see iTerm2 does reset the palette on exit.

typescript.txt

https://user-images.githubusercontent.com/71736629/135063762-25259f92-30f9-4de9-9a88-addf07ac4d6a.mp4

[nicm]

When it is working, is the palette being reset to the defaults, or to the palette you have set beforehand with some other method?

[nicm]

I do not see anything in this file that will reset the palette to the defaults and AFAIK there is no mechanism at all to reset it to the previous state before any particular point. Indeed when I cat the file, the palette is not reset.

iTerm2 does not appear to reset the palette on RIS (\033c) or with the "reset" menu option, nor can I see any way in the code to cause it to reset other than sending OSC 104.

Are you sure that your custom palette (whatever it is) is not being reapplied by your shell or something else?

[andmarti1424]

@nicm Please look at the screencast, this problem does not occur in most terminals (accept Alacritty) when sc-im is not called from within tmux, but always happens when sc-im is called from within tmux. It makes me wonder if this is because tmux's pane-colours and sc-im's DEFINE_COLOR both update the same library that defines tmux's colors & if anything can be done about it. @ThomasDickey's inputs will be appreciated too, as @andmarti1424 suspects this has something to do with ncurses.

If this fails on some terminals and others not, then the issue might be in it (or tmux) and not ncurses.

[poetaman]

@andmarti1424 Not sure if this will get resolved anytime soon. In my testing, the problem occurs only when user is defining some color and using it. To bypass the problem, would it be possible to give user the ability to specify 256 colors? Like COLOR0-COLOR255 or COLOR16-COLOR255 given 16 lower colors are already covered? Thanks.

[andmarti1424]

@reportaman I really dont know. As I said before, it seems an ncurses or tmux issue. Certainly not an issue of sc-im. How could more colors bypass the issue u say?

[poetaman]

@andmarti1424 The problem occurs only if the user tries to define a color, which corrupts some state somewhere. Currently, one can use 16 colors without the need for defining any (RED, BLUE, etc). If user is able to access the default 256 colors, it will just enlarge their repertoire of colors without running into this issue, which we don't know the origin of.

As a sidenote: After seeing the benefits of base-16 colors I have decided to stick with them. That way, if the terminal colorscheme is changed, it automatically reflects in the colorscheme of sc-im application. Such dynamic update of colors is not possible if one defines or redefines a color in scimrc and uses it. Its paradoxical, by limiting myself to 16 colors, I actually get hundreds of sc-im colorschemes for free.

https://user-images.githubusercontent.com/71736629/153660892-155ab74e-cf4c-4382-9da4-918e1c1712ed.mp4