Menus not rendering as expected

[DanLipsitt]

Small description Menus look strange. See screenshot below. I also sometimes get junk characters left behind when menus close, but I don't have a repro for that.

Actual result with screenshot

Steps to reproduce with sample data and a .vd Open any menu at any time. Window size and font don't seem to matter. I've tried:

• MesloLGS NF (a fancy variant of Menlo)
• Menlo (typical Mac default )
• Courier

Additional context Visidata: saul.pw/VisiData v2.8 iTerm2: 3.4.12 or Terminal.app: 2.11 MacOS: 11.2.3

Let me know if there are any terminal parameters or env vars, etc that might help diagnose this.

[saulpw]

Wow, that is super-weird, @DanLipsitt! I haven't seen anything like this before. We'll have to get a Mac and see if we can reproduce this somehow. Please do let us know if you figure out anything that fixes it.

[DanLipsitt]

GitHub and CircleCI both have MacOS runners. Maybe you can compare against Linux output with some sort of snapshot testing. I noticed Intro to VisiData has text-only "screenshots." They appear to be created with Termscript. Would that be a sensible approach? Snapshots could catch future regressions but I guess that reproducing this current bug wouldn't help unless there is a previous working version and we can bisect the git history.

[jsvine]

@DanLipsitt Hiya, I'm the author of that tutorial. As it happens, this is the first time I'm seeing the Termscript repo you've linked to — interesting! The "termscript" referenced in the tutorial's build scripts actually reference a local, non-public library I built. (The idea, however, appears to be pretty similar, and I'm not surprised we chose the same name, which is somewhat obvious.) I've thought about open-sourcing it, but haven't found the time to properly document it or add proper tests, et cetera.

[saulpw]

@jsvine do you see the menu weirdness that @DanLipsitt is seeing on MacOS?

[jsvine]

Hmmm, I do not.

[daviewales]

I'm on macOS 10.13 with iTerm 3.3.12 and MesloLGS NF font, and I can't reproduce. Also tried in Terminal 2.8.3, and can't reproduce.

[nsmaciej]

I'm also having this issue. iTerm2 3.5.0beta4, macOS 12.1, vd 2.8, SF Mono Regular 12pt

[frosencrantz]

Is the problem in VisiData, Python, Curses or is it in the display iTerm2, font, MacOs?
Could you try running VisiData in asciinema and share the result?

That would capture the output from VisiData, and could help tell us which side the problem is on. https://asciinema.org/docs/installation

[nsmaciej]

Sure, here you go. There is definitely something interesting going on judging by this. It actually looked so bad I made another recording where I run htop just to make asciinema works.

[nsmaciej]

Might be a recent iTerm2 issue actually? I swear I tried it before in Terminal.app when I first saw this but checking it now it seems to work perfectly there as well as the current stable release of iTerm2 (3.4.15).

It's broken in iTerm2 3.5.0beta4 and 3.5.20220107-nightly (I checked with the stock preferences just in case).

Here are the screenshots of it working on Terminal.app on macOS 12.1 (21C52) and iTerm2 3.4.15 with the same config as my original screenshot. Sure enough the asciinema on 3.4.15 looks perfect. Might be worthwhile trying to bisecting through iTerm2 releases between the author's version (3.4.12) and 3.4.15.

[nsmaciej]

Hmm... I told myself I won't spend more time on this today, but couldn't resist checking author's version. I actually can't reproduce it on 3.4.12, so from my point of view it looks like a beta-only issue, but I can't explain the original bug.

[frosencrantz]

Glad it is working now. The first 2 asciinema recordings show the issue, which looks like the bug might have been on the VisiData/Python/Curses side. Though the colors look very dark, maybe this is a configuration issue. Did you also update Python or VisiData or your .visidatarc file?

Also wondering if @DanLipsitt is still having the issue?

[nsmaciej]

Homebrew VisiData seems to use Python 3.9.9 right now and I am not sing a .visidatarc file. I have no idea why the colours look the way they do in the recordings, because in person VisiData is mostly usable on the nightly/beta iTerm2 other than the menu glitches.

[anjakefala]

Based on the asciinema output, and the versions involved, my current hypothesis is that this issue is describing the problem: https://gitlab.com/gnachman/iterm2/-/issues/9945.

I am going to comment on that issue with the iTerm2 folks, and if they respond I can contribute that information to the bug.

@DanLipsitt does upgrading your version of iTerm2 to the latest stable resolve the issue for you?

[anjakefala]

It seems like this bug is connected to a rendering bug in a particular version of iTerm2.

If someone else runs into this, I recommending upgrading iTerm2 to at least 3.4.15. If you are still running into the issue, then please comment on this issue!

[DanLipsitt]

This is not limited to iTerm2. Here it is on MacOS Terminal.app 2.11: I am also still seeing this on iTerm2 3.4.12 (which seems to be the latest for Arm).

[pnfnp]

On iTerm Build 3.4.15 using fonts Monaco and Hack Nerd Fonts is working just fine 👌

[DanLipsitt]

I tried @pnfnp's combo of iTerm 3.4.15 and Monaco and still experienced the bug.

[anjakefala]

Hello everyone!

None of the core maintainers use MacOS, so we feel a little stuck with this issue. Since none of the core maintainers can make progress on it, we are going to tag it as an "unsolved mystery", and close it.

We welcome folks to continue sharing ideas, and if anyone does want to adopt getting to the bottom of it, we can re-open.

Sorry, @DanLipsitt, I do appreciate that you tried the different version and font.

[dufferzafar]

I can confirm that this issue is not specific to MacOS. I'm using Visidata 2.8 on MATE Linux and am seeing essentially the same behaviour, where the menu doesn't render properly.

This is the case with both Konsole & Kitty terminals:

@saulpw @anjakefala Please let me know if you want me to provide any information to help debug this.

[anjakefala]

@dufferzafar Welp!

Which font are you using?

[saulpw]

Everyone who is experiencing this bug: what version of Python are you using?

[nsmaciej]

I'm still experiencing the issue in iTerm 3.5.0beta5. The homebrew version of visidata I'm using uses Python 3.9.9

[DanLipsitt]

Still buggy in iTerm2 and Terminal.app. Visidata 2.8 installed with pipx on python 10.0.0.

[lkhrs]

I can't reproduce the issues with Terminal.app and iTerm2. I'm using VisiData 2.8 on macOS 12.4. I tried changing the font, the font sizes, and tried using the default out-of-the-box profile.

I was able to reproduce the issue with kitty, WezTerm, and Alacritty. Like iTerm and Terminal.app, changing the font or the font size had no effect.

Clockwise, from top left: Alacritty 0.10.1 with Hack Nerd Font Mono, WezTerm 20220624 with JetBrains Mono + its own set of glyphs, iTerm2 3.4.16 with JetBrains Mono and Meslo for glyphs, Terminal.app 2.12.7 with Meslo, kitty 0.25.2 with Meslo, and cool-retro-term with Terminus.

[lkhrs]

Update: I was able to fix the issue in kitty, Alacritty, and WezTerm by setting their $TERM value to xterm or xterm-256color.

I noticed that the only 3 terminals with issues in the screenshot above also used custom values for $TERM, like xterm-kitty, alacritty, and wezterm respectively.

In Terminal.app and iTerm2, the default is xterm-256color.

I observed that running vd in tmux also works, even if running vd directly in the terminal shows the glitch. tmux reports $TERM as screen-256color.

[dmd]

I still have this bug in iTerm2 even with exporting TERM=xterm or TERM=xterm-256color

[saulpw]

@dmd Please include versions of your terminal, font, etc. We need a repro for ourselves before we can have a hope of fixing it.

[dmd]

In super bizarre news, now I can't reproduce it - after a reboot. Which is not something that could possibly have helped. But there you go.

[saulpw]

Well, glad it's fixed for you, anyway! :)

[mickronome]

Stumbled upon this issue just now using TERM="xterm-256color".

But since I had recently been doing some weird scripts with infocmp, I had a hunch.

iTerm2 (and I suspect others) seems to set then environment variable TERMINFO_DIRS to (in my case) TERMINFO_DIRS="/Applications/iTerm.app/Contents/Resources/terminfo:/usr/share/terminfo" to pull in their own variants of the terminfo files.

Though infocmp didn't report any differences that screamed "bug" between the system and iTerm2 provided terminfo files, setting TERMINFO_DIRS to /usr/share/terminfo to sidestep the xterm-256color terminfo provided by iTerm2 resolved the issue, and setting TERMINFO_DIRS to only use the iTerm2 provided files reintroduced issue.

A simple workaround for occasional use/testing is to run TERMINFO_DIRS=/usr/share/terminfo vd filename.csv.

Similarly TERMINFO_DIRS=/usr/share/terminfo infocmp | head -n 1 on macOS should print something like: # Reconstructed via infocmp from file: /usr/share/terminfo/78/xterm-256color if $TERM is set to xterm-256color.

Happy bug hunting...

ALT_DIR=/Applications/iTerm.app/Contents/Resources/terminfo/
SYS_DIR=/usr/share/terminfo/
TERMID="xterm-256color"
echo
echo "------------------ COMPARE --------------------------------------------------------------"
infocmp -A "$SYS_DIR" -B "$ALT_DIR" "$TERMID" "$TERMID"

echo
echo "------------------ COMPARE (include commented out capabilities) --------------------------"
infocmp -a -A "$SYS_DIR" -B "$ALT_DIR" "$TERMID" "$TERMID"

------------------ COMPARE --------------------------------------------------------------
comparing xterm-256color to xterm-256color.
    comparing booleans.
    comparing numbers.
    comparing strings.
        dim: NULL, '\E[2m'.
        ka1: NULL, '\EOw'.
        ka3: NULL, '\EOy'.
        kb2: '\EOE', '\EOu'.
        kc1: NULL, '\EOq'.
        kc3: NULL, '\EOs'.
        mgc: NULL, '\E[?69l'.
        oc: NULL, '\E]104\007'.
        rep: NULL, '%p1%c\E[%p2%{1}%-%db'.
        ritm: NULL, '\E[23m'.
        rmcup: '\E[?1049l', '\E[?1049l\E[23;0;0t'.
        rs1: '\Ec', '\Ec\E]104\007'.
        setab: '\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48;5;%p1%d%;m', '\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48\:5\:%p1%d%;m'.
        setaf: '\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38;5;%p1%d%;m', '\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38\:5\:%p1%d%;m'.
        setb: NULL, '\E[4%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m'.
        setf: NULL, '\E[3%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m'.
        sgr: '%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m', '%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p5%t;2%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m'.
        sitm: NULL, '\E[3m'.
        smcup: '\E[?1049h', '\E[?1049h\E[22;0;0t'.
        smglr: NULL, '\E[?69h\E[%i%p1%d;%p2%ds'.
        u8: '\E[?1;2c', '\E[?%[;0123456789]c'.

------------------ COMPARE (include commented out capabilities) --------------------------
comparing xterm-256color to xterm-256color.
    comparing booleans.
        XT: F:T.
    comparing numbers.
    comparing strings.
        dim: NULL, '\E[2m'.
        ka1: NULL, '\EOw'.
        ka3: NULL, '\EOy'.
        kb2: '\EOE', '\EOu'.
        kc1: NULL, '\EOq'.
        kc3: NULL, '\EOs'.
        mgc: NULL, '\E[?69l'.
        oc: NULL, '\E]104\007'.
        rep: NULL, '%p1%c\E[%p2%{1}%-%db'.
        ritm: NULL, '\E[23m'.
        rmcup: '\E[?1049l', '\E[?1049l\E[23;0;0t'.
        rs1: '\Ec', '\Ec\E]104\007'.
        setab: '\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48;5;%p1%d%;m', '\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48\:5\:%p1%d%;m'.
        setaf: '\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38;5;%p1%d%;m', '\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38\:5\:%p1%d%;m'.
        setb: NULL, '\E[4%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m'.
        setf: NULL, '\E[3%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m'.
        sgr: '%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m', '%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p5%t;2%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m'.
        sitm: NULL, '\E[3m'.
        smcup: '\E[?1049h', '\E[?1049h\E[22;0;0t'.
        smglr: NULL, '\E[?69h\E[%i%p1%d;%p2%ds'.
        u8: '\E[?1;2c', '\E[?%[;0123456789]c'.
        Cr: NULL, '\E]112\007'.
        Cs: NULL, '\E]12;%p1%s\007'.
        E3: NULL, '\E[3J'.
        Ms: NULL, '\E]52;%p1%s;%p2%s\007'.
        Se: NULL, '\E[2 q'.
        Setulc: NULL, '\E[58\:\:2\:\:%p1%{65536}%/%d\:\:%p1%{256}%/%{255}%&%d\:\:%p1%{255}%&%d%;m'.
        Smulx: NULL, '\E[4\:%p1%dm'.
        Ss: NULL, '\E[%p1%d q'.
        XM: NULL, '\E[?1000%?%p1%{1}%=%th%el%;'.
        ka2: NULL, '\EOx'.
        kb1: NULL, '\EOt'.
        kb3: NULL, '\EOv'.
        kc2: NULL, '\EOr'.
        kp5: NULL, '\EOE'.
        kpADD: NULL, '\EOk'.
        kpCMA: NULL, '\EOl'.
        kpDIV: NULL, '\EOo'.
        kpDOT: NULL, '\EOn'.
        kpMUL: NULL, '\EOj'.
        kpSUB: NULL, '\EOm'.
        kpZRO: NULL, '\EOp'.
        rmxx: NULL, '\E[29m'.
        smxx: NULL, '\E[9m'.
        xm: NULL, '\E[M%?%p4%t%p3%e%{3}%;%' '%+%c%p2%'!'%+%c%p1%'!'%+%c'.

[9mm]

I also have this problem on latest iterm...

what can i provide to debug this?

[9mm]

it also persists across reboots for me

[pnfnp]

I'm facing the same issue with iTerm2, but for those using tmux you can bypass the issue and render correctly.

iTerm2

tmux

[lkhrs]

Regarding iTerm2, I did some digging and the issue appears to be affected by the $TERMINFO_DIRS setting under Advanced > Scripting. Setting this to “No” fixes the issue in iTerm2 build 3.5.0.

As to which terminfo feature is causing the problem, I’m not sure. I tracked down the source directory that gets loaded into the $TERMINFO_DIRS variable: https://gitlab.com/gnachman/iterm2/-/tree/master/Resources/terminfo/78

VisiData v2.11,1

Edit: I re-read the thread and mickronome already tracked the cause down to terminfo. Leaving this up to point out a GUI option for disabling iTerm2’s extra terminfo features.

[9mm]

@lkhrs do you know any downsides to disabling that, or what features i miss out on?

[rpuncel]

Regarding iTerm2, I did some digging and the issue appears to be affected by the $TERMINFO_DIRS setting under Advanced > Scripting. Setting this to “No” fixes the issue in iTerm2 build 3.5.0.

This fixed histogram characters that were printing as unknown characters for me! Leaving some details in case this helps anyone else

• iTerm2 version: Build 3.5.0beta22
• ZSH shell with powerlevel10k theme
• Font: MesloLGS NF

Before ("Set $TERMDIRINFO_DIRS..." set set to true):

After ("Set $TERMDIRINFO_DIRS..." set to false: