search

tenox7 / ttyplot

a realtime plotting utility for terminal/console with data input from stdin
Apache License 2.0
1.15k stars 48 forks source link

[1.6.1] Chart output corrupts after a few minutes on macOS? #166

Open quinncomendant opened 10 months ago

quinncomendant commented 10 months ago

I recently upgraded ttyplot on macOS from 1.5.2 to 1.6.1 (via brew upgrade), and the new version displays some misplaced character artifacts in the chart after a few minutes of running. I use this command:

ping -i .1 8.8.8.8 | sed -u 's/^.*time=//g; s/ ms//g; s/^.*timeout.*$//g' | ttyplot -t "ping to 8.8.8.8" -u ms

I've run the first part of the command simutaneously in a separate window to view the input data to ensure it is not causing ttyplot to become corrupted, and it does not: each line contains just one numeric value.

The chart starts out displaying correctly:

Screenshot 2024-01-17 at 19 39 26

But after a few minutes, once the variation in values changes enough for ttyplot to rescale the chart, it can look like this:

Screenshot 2024-01-17 at 19 56 03

Since this was not happening with ttyplot version 1.5.2, it must be something with the new version (which has a new plotting algorithm?).

hartwork commented 10 months ago

Hi @quinncomendant, thanks for the report! I do not have access to macOS outside of CI personally, running your command for 5 minutes (with multiple re-scales) on Linux does not show anything off for me. Is there a chance that you could:

quinncomendant commented 10 months ago

Hi @hartwork! Thanks for the quick reply with suggestions. Yeah, if your internet is stable, ping may not reveal the issue.

In order to reproduce this, you need to generate values that:

Here's a test case you can use to reproduce the issue:

  1. Set your terminal window to about 90 columns by 20 rows.
  2. Run this:
for i in 13.669 13.669 13.669 17.235 17.235 17.235 17.235 17.235 17.235 17.235 10.831 10.831 10.831 10.831 10.831 10.831 93.902 93.902 93.902 93.902 93.902 93.902 93.902 93.902 2.984 2.984 2.984 2.984 2.984 2.984 2.984 6.528 6.528 6.528 6.528 6.528 6.528 6.528 6.528 5.143 5.143 5.143 5.143 5.143 5.143 5.143 5.143 8.741 8.741 8.741 8.741 8.741 8.741 8.741 8.741 2.310 2.310 2.310 2.310 2.310 2.310 2.310 10.943 10.943 10.943 10.943 10.943 10.943 10.943 10.943 18.117 18.117 18.117 18.117 18.117 18.117 18.117 18.117 11.670 11.670 11.670 11.670 11.670 11.670 11.670 11.670 5.296 5.296 5.296 5.296 5.296 5.296 5.296 5.296 8.840 8.840 8.840 8.840 8.840 8.840 8.840 8.840 7.446 7.446 7.446 7.446 7.446 7.446 7.446 7.446 7.446 6.027 6.027 6.027 6.027 6.027 6.027 6.027 6.027 4.598 4.598 4.598 4.598 4.598 4.598 4.598 4.598 4.598 8.156 8.156 8.156 8.156 8.156 8.156 8.156 8.156 8.156 6.724 6.724 6.724 6.724 6.724 6.724 6.724 6.724 5.336 5.336 5.336 5.336 5.336 5.336 5.336 5.336 5.336 8.920 8.920 8.920 8.920 8.920 8.920 8.920 8.920 2.485 2.485 2.485 2.485 2.485 2.485 2.485 2.485 2.485 6.056 6.056 6.056 6.056 6.056 6.056 6.056 6.056 3.284 3.284 3.284 3.284 3.284 3.284 3.284 3.284 6.840 6.840 6.840 6.840 6.840 6.840 6.840 6.840 6.840 5.382 5.382 5.382 5.382 5.382 5.382 5.382 5.382 3.986 3.986 3.986 3.986 3.986 3.986 3.986 3.986 3.986 2.608 2.608 2.608 2.608 2.608 2.608 2.608 2.608 1.134 1.134 1.134 1.134 1.134 1.134 1.134 1.134 1.134 4.777 4.777 4.777 4.777 4.777 4.777 4.777 4.777 3.337 3.337 3.337 3.337 3.337 3.337 3.337 3.337 3.337 6.924 6.924 6.924 6.924 6.924 6.924 6.924 6.924 50.511 50.511 50.511 50.511 50.511 50.511 50.511 50.511 50.511 4.095 4.095 4.095 4.095 4.095 4.095 4.095 4.095 2.677 2.677 2.677 2.677 2.677 2.677 2.677 2.677 2.677 1.278 1.278 1.278 1.278 1.278 1.278 1.278 1.278 3.443 3.443 3.443 3.443 3.443 3.443 3.443 3.443 3.443 7.014 7.014 7.014 7.014 7.014 7.014 7.014 7.014 5.616 5.616 5.616 5.616 5.616 5.616 5.616 5.616 5.616 7.731 7.731 7.731 7.731 7.731 7.731 7.731 7.731 7.731 6.371 6.371 6.371 6.371 6.371 6.371 6.371 6.371 4.914 4.914 4.914 4.914 4.914 4.914 4.914 4.914 4.914 3.492 3.492 3.492 3.492 3.492 3.492 3.492 3.492 2.086 2.086 2.086 2.086 2.086 2.086 2.086 2.086 2.086 0.660 0.660 0.660 0.660 0.660 0.660 0.660 0.660 4.234 4.234 4.234 4.234 4.234 4.234 4.234 4.234 4.234 2.853 2.853 2.853 2.853 2.853 2.853 2.853 2.853 2.853 1.382 1.382 1.382 1.382 1.382 1.382 1.382 1.382 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 8.607 8.607 8.607 8.607 8.607 8.607 8.607 8.607 2.206 2.206 2.206 2.206 2.206 2.206 2.206 2.206 2.206 5.747 5.747 5.747 5.747 5.747 5.747 5.747 5.747 7.980 7.980 7.980 7.980 7.980 7.980 7.980 7.980 6.509 6.509 6.509 6.509 6.509 6.509 6.509 6.509 6.509 5.088 5.088 5.088 5.088 5.088 5.088 5.088 5.088 5.088 3.682 3.682 3.682 3.682 3.682 3.682 3.682 3.682 2.264 2.264 2.264 2.264 2.264 2.264 2.264 2.264 2.264 5.868 5.868 5.868 5.868 5.868 5.868 5.868 5.868 4.424 4.424 4.424 4.424 4.424 4.424 4.424 4.424 4.424 7.988 7.988 7.988 7.988 7.988 7.988 7.988 7.988 6.607 6.607 6.607 6.607 6.607 6.607 6.607 6.607 6.607 5.181 5.181 5.181 5.181 5.181 5.181 5.181 5.181 8.746 8.746 8.746 8.746 8.746 8.746 8.746 8.746 8.746; do echo $i; sleep 0.1; done | ttyplot

This prints a value to the screen every 1/10 seconds. After about 12 seconds, the screen will look like this:

Screenshot 2024-01-18 at 19 55 27

hartwork commented 10 months ago

Hi @quinncomendant, thanks for the update! Neither after 12 seconds nor anywhere between start and finish anything like that appears for me with 1.6.1 on (Gentoo) Linux, my terminal was exactly 90x20 in Terminator. Could you try with Linux or from source or on different (macOS) hardware?

hartwork commented 10 months ago

@quinncomendant update: no corruptions like that with 90x20 in Terminator on Debian bookworm with ttyplot 1.6.1 built from source, either.

edgar-bonet commented 10 months ago

Hi @quinncomendant, and thanks for the report. I couldn't reproduce the issue either. I am using:

quinncomendant commented 10 months ago

I did more experiments. The issue only occurs on macOS, when using iTerm version 3.5.0beta18, running ttyplot v1.6.1.

I know terminal emulation can be difficult to troubleshoot because of all the variables on multiple layers. I would be happy to just blame iTerm because it's a huge ball of wax. However, the fact that ttyplot v1.5.2 works, and v1.6.1 does not, in identical environments, is evidence that something changed that made ttyplot less resilient to working well in obscure environments, namely, mine.

Here's a screen recording showing v1.5.2 and v1.6.1 running side-by-side.

hartwork commented 10 months ago

@quinncomendant thanks a lot for investigating further and sharing your findings and the recording! My personal key takeway from what you shared is that the combination of local operation + iTerm2 + ttyplot 1.6.1 is the problem. (I'm assuming here that you already are running the latest version 3.4.23 of iTerm2 so that we're not dealing with already fixed bugs in iTerm, either something novel or outside of iTerm2?) To me it sounds like the local NCurses layer could be involved (because with SSH part of NCurses is no longer local). There were two bigger pull requests between 1.5.1 and 1.6.0 that changed things in the NCurses area:

What I am guessing right now is the symptom would go away adding (or re-adding) "full-screen" redrawing somewhere. I'm ignoring the question here whether ttyplot or iTerm would be at fault then then but that's at least an area where things changed and where we could experiment to see if it makes the symptoms go away. The pull requests are not small and not trivial but maybe you'd still see something that stands out to you as a candidate for the source of that problem. Could you have a quick look at the two and see if something catches your intention for a potential cause or for something with potential for experimentation?

edgar-bonet commented 10 months ago

@quinncomendant: Some random ideas...

quinncomendant commented 9 months ago

I tested this again on an older MacBook which I recently reinstalled the OS, so it has a virgin terminal environment (never used, never customized). I installed iTerm and ttyplot for the first time. The issue occurs there!

Also, I discovered it only occurs with the beta version of iTerm (3.5.0beta18), and not the stable release (3.4.23).

The beta version of iTerm is far ahead of the stable release, so I've been using it everywhere. I think the iTerm team even recommends it. Searching the changelog of the beta releases I can see many dozens of fixes related to drawing text and screen management, so it's no surprise that there is a difference.

@hartwork, I checked those two pull requests and briefly scanned the commits, but I'm not familiar enough with ttyplot's design to know which might be more the cause. If I had to guess on a hunch, it seems like c150198 has the most changes related to screen redraw, so we could start there? If there any way I can get a copy of the source code before these pull requests and merge them myself? (I'm not very familiar with GitHub.)

@edgar-bonet, I can do the random ideas you suggested, but I think it may not be useful since I already discovered ttyplot 1.6.1 does work fine locally on macOS when using the default Terminal.app or the stable version of iTerm, it only occurs with the beta version of iTerm. It's not environmental differences that cause the issue.

edgar-bonet commented 9 months ago

@quinncomendant:

If there any way I can get a copy of the source code before these pull requests and merge them myself?

The Web UI way:

The git way:

git clone https://github.com/tenox7/ttyplot.git
cd ttyplot
git checkout c150198^  # the “^” suffix means “parent of”

Note that git can be very helpful for bisecting the history and get the exact commit that broke something:

git bisect start
git bisect good 1.5.2
git bisect bad 1.6.1

At this point git checks out an intermediate revision and asks you to test it. You reply by typing either git bisect good or git bisect bad. Repeat. After O(log(n)) steps, git tells you the ID of the commit that caused the breakage.

It's not environmental differences that cause the issue.

That's not certain. Ttyplot relies on ncurses, and ncurses uses the TERM environment variable and the terminfo database to know what escape sequences it has to give to the terminal to get a specific result. Terminal.app uses a different terminfo entry from iTerm. When you ssh to Ubuntu, ttyplot uses the terminfo database installed on the Ubuntu server, instead of the one on your Mac. If the iTerm entries in these databases differ, that could lead to different behaviors.

hartwork commented 9 months ago

@edgar-bonet suggesting git bisect is a great idea, didn't think of in this context yet, but should have.

@quinncomendant one thing I'm wondering is if 1.5.2 works for you and 1.6.1 how did you install each and where do they get ther (N)Curses from — from Homebrew or from stock macOS? Both of these versions? I see potential for more insights and variance there, maybe it can be pinned down further. To re-summarize, so far we have that it needs all of:

So (1) from source and/or homebrew and (2) the precise source and version of (N)Curses could give additional insights.

quinncomendant commented 9 months ago

@edgar-bonet Thanks for the tip to use git bisect, that's super cool! The process of bisecting and testing went smoothly until a commit that failed to compile. I was unable to git bisect skip because it was the last commit that needed testing. Here's the output at the end of the git bisect process:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
c73270ccfb74cf2dddfe9526f68645630b43f9f5
61c2cf5ccc4a951d5b6bf4f9caf7ee5f75c64d3e
We cannot bisect more!

I manually checkout’ed a series of commits in this range and tested them one-by-one, with the following results (in order from newest commits to older):

This is confirms what git bisect already said, that the bug was introduced in c73270c or 61c2cf5.

It's not environmental differences that cause the issue.

That's not certain. Ttyplot relies on ncurses, and ncurses uses the TERM environment variable […]

Throughout my testing, $TERM has always been xterm-256color everywhere. I still don't understand how my environment could be different between two ttyplot versions, but maybe I missed your point.

@hartwork As far as I can tell there is only one ncurses installed: the one included with the macOS developer tools. According to otool -L, all the versions I've tested use the same ncurses library, including the one installed by Homebrew (which just uses make during its install process).

❯ ./ttyplot -v
ttyplot 1.5.2

❯ otool -L ./ttyplot
./ttyplot:
    /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
❯ /opt/homebrew/bin/ttyplot -v
ttyplot 1.6.1

❯ otool -L /opt/homebrew/bin/ttyplot
/opt/homebrew/bin/ttyplot:
    /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
hartwork commented 9 months ago

Hi @quinncomendant, thanks for investigating more and sharing your findings with us! I believe what you found means the answer may be be somewhere in c73270ccfb74cf2dddfe9526f68645630b43f9f5, which notably switches from ncurses to ncursesw. I'm surprised that ttyplot links to /usr/lib/libncurses.5.4.dylib without any "w" in it on your system. Could you share the output of pkg-config --cflags ncursesw ; pkg-config --libs ncursesw?

If we're looking at some flavor of memory corruption, maybe compiling with AddressSanitizer and UndefinedBehaviorSanitizer and running again could lead to insights:

env {C,LD}FLAGS=-fsanitize=address,undefined make clean all
[..data piping here..] | MallocNanoZone=0 ./ttyplot [..arguments here..]

If it now crashes at runtime, that would be a first clue.

@edgar-bonet any ideas what in c73270ccfb74cf2dddfe9526f68645630b43f9f5 it could possible be (if anything)? I remember because of the global static we were sure that all memory is zero-initialized. Anything else it could possible be?

edgar-bonet commented 9 months ago

@quinncomendant: Thanks for bisecting this! It is really useful.

Commit 61c2cf5ccc4a951d5b6bf4f9caf7ee5f75c64d3e is only about getting the -I compiler flags right in order for the program to build. The issue is thus almost certainly triggered by c73270ccfb74cf2dddfe9526f68645630b43f9f5. This is a quite disruptive commit: we move from the historical ASCII-only ncurses API to to newer (tough probably still ancient) “wide character” API.

I did a small test to capture the output of ttyplot:

script -c 'seq 1 10 | ./ttyplot'

and compared c73270c with its parent. The most striking difference is that the output was initially ASCII-only, with lines made out of DEC Special Graphics, which got replaced by UTF-8-encoded Box-drawing characters. For example, the sequence

ESC ( 0 ESC [ 0 m x ESC ( B

where the x encodes a vertical line, became just │ (U+2502 Box drawings light vertical, encoded in UTF-8 as 0xe2 0x94 0x82).

However, since the vertical lines are initially rendered correctly, this difference in output does not explain the problem.

Throughout my testing, $TERM has always been xterm-256color everywhere.

You may run

infocmp xterm-256color

on both your Mac and the Ubuntu server, and compare the outputs.

@hartwork wrote:

I'm surprised that ttyplot links to /usr/lib/libncurses.5.4.dylib without any "w" in it on your system.

We have seen that the ASCII-only and the wide-character-capable APIs are declared in the same header file... or in different header files, depending on the OS. There seems to be quite a bit of variability in how ncurses is deployed. Since the wide-character-capable API is a superset of the older one, I am not that much surprised that a system could provide only one dynamic library with the most complete API.

quinncomendant commented 9 months ago

@hartwork,

Could you share the output of pkg-config --cflags ncursesw ; pkg-config --libs ncursesw?

❯ pkg-config --cflags ncursesw

❯ pkg-config --libs ncursesw
-lncurses

env {C,LD}FLAGS=-fsanitize=address,undefined make clean all [..data piping here..] | MallocNanoZone=0 ./ttyplot [..arguments here..]

Building and running it like that changed nothing.

quinncomendant commented 9 months ago

@edgar-bonet,

[…] which got replaced by UTF-8-encoded Box-drawing characters […]

Ah yes, this is interesting.

Another possible clue: the issue can be reproduced simply by resizing the window, but only after the corruption occurs while when rendering the plot. Corruption only occurs if the set of values is widely variable such that the chart needs to rescale after some very large numbers scroll off the screen. If a plot is rendered with consistent values (so the plot never needs to rescale), the screen never corrupts, and then the issue does not present itself by resizing the window. But once the corruption has occurred, even very briefly, resizing the window will consistently cause display issues.

So, the UTF box-drawing characters might not be to blame (since their presence does not, on its own, cause the corruption), but perhaps only in relation to the chart rescale algorithm.

You may run infocmp xterm-256color

Oh, wow, more dark secrets of the terminal. 🙃

Indeed, there is a difference between Terminal.app and iTerm stable, and iTerm beta:

Terminal.app and iTerm stable (no corruption)

#   Reconstructed via infocmp from file: /usr/share/terminfo/78/xterm-256color
xterm-256color|xterm with 256 colors,
    am, bce, ccc, km, mc5i, mir, msgr, npc, xenl,
    colors#256, cols#80, it#8, lines#24, pairs#32767,
    acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
    bel=^G, blink=\E[5m, bold=\E[1m, cbt=\E[Z, civis=\E[?25l,
    clear=\E[H\E[2J, cnorm=\E[?12l\E[?25h, cr=^M,
    csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
    cud=\E[%p1%dB, cud1=^J, cuf=\E[%p1%dC, cuf1=\E[C,
    cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,
    cvvis=\E[?12;25h, dch=\E[%p1%dP, dch1=\E[P, dim=\E[2m,
    dl=\E[%p1%dM, dl1=\E[M, ech=\E[%p1%dX, ed=\E[J, el=\E[K,
    el1=\E[1K, flash=\E[?5h$<100/>\E[?5l, home=\E[H,
    hpa=\E[%i%p1%dG, ht=^I, hts=\EH, ich=\E[%p1%d@,
    il=\E[%p1%dL, il1=\E[L, ind=^J, indn=\E[%p1%dS,
    initc=\E]4;%p1%d;rgb\:%p2%{255}%*%{1000}%/%2.2X/%p3%{255}%*%{1000}%/%2.2X/%p4%{255}%*%{1000}%/%2.2X\E\\,
    invis=\E[8m, is2=\E[!p\E[?3;4l\E[4l\E>, kDC=\E[3;2~,
    kEND=\E[1;2F, kHOM=\E[1;2H, kIC=\E[2;2~, kLFT=\E[1;2D,
    kNXT=\E[6;2~, kPRV=\E[5;2~, kRIT=\E[1;2C, kb2=\EOE, kbs=^H,
    kcbt=\E[Z, kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA,
    kdch1=\E[3~, kend=\EOF, kent=\EOM, kf1=\EOP, kf10=\E[21~,
    kf11=\E[23~, kf12=\E[24~, kf13=\E[1;2P, kf14=\E[1;2Q,
    kf15=\E[1;2R, kf16=\E[1;2S, kf17=\E[15;2~, kf18=\E[17;2~,
    kf19=\E[18;2~, kf2=\EOQ, kf20=\E[19;2~, kf21=\E[20;2~,
    kf22=\E[21;2~, kf23=\E[23;2~, kf24=\E[24;2~,
    kf25=\E[1;5P, kf26=\E[1;5Q, kf27=\E[1;5R, kf28=\E[1;5S,
    kf29=\E[15;5~, kf3=\EOR, kf30=\E[17;5~, kf31=\E[18;5~,
    kf32=\E[19;5~, kf33=\E[20;5~, kf34=\E[21;5~,
    kf35=\E[23;5~, kf36=\E[24;5~, kf37=\E[1;6P, kf38=\E[1;6Q,
    kf39=\E[1;6R, kf4=\EOS, kf40=\E[1;6S, kf41=\E[15;6~,
    kf42=\E[17;6~, kf43=\E[18;6~, kf44=\E[19;6~,
    kf45=\E[20;6~, kf46=\E[21;6~, kf47=\E[23;6~,
    kf48=\E[24;6~, kf49=\E[1;3P, kf5=\E[15~, kf50=\E[1;3Q,
    kf51=\E[1;3R, kf52=\E[1;3S, kf53=\E[15;3~, kf54=\E[17;3~,
    kf55=\E[18;3~, kf56=\E[19;3~, kf57=\E[20;3~,
    kf58=\E[21;3~, kf59=\E[23;3~, kf6=\E[17~, kf60=\E[24;3~,
    kf61=\E[1;4P, kf62=\E[1;4Q, kf63=\E[1;4R, kf7=\E[18~,
    kf8=\E[19~, kf9=\E[20~, khome=\EOH, kich1=\E[2~,
    kind=\E[1;2B, kmous=\E[M, knp=\E[6~, kpp=\E[5~,
    kri=\E[1;2A, mc0=\E[i, mc4=\E[4i, mc5=\E[5i, meml=\El,
    memu=\Em, op=\E[39;49m, rc=\E8, rev=\E[7m, ri=\EM,
    rin=\E[%p1%dT, ritm=\E[23m, rmacs=\E(B, rmam=\E[?7l,
    rmcup=\E[?1049l, rmir=\E[4l, rmkx=\E[?1l\E>,
    rmm=\E[?1034l, rmso=\E[27m, rmul=\E[24m, rs1=\Ec,
    rs2=\E[!p\E[?3;4l\E[4l\E>, sc=\E7,
    setab=\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48;5;%p1%d%;m,
    setaf=\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38;5;%p1%d%;m,
    sgr=%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p5%t;2%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m,
    sgr0=\E(B\E[m, sitm=\E[3m, smacs=\E(0, smam=\E[?7h,
    smcup=\E[?1049h, smir=\E[4h, smkx=\E[?1h\E=,
    smm=\E[?1034h, smso=\E[7m, smul=\E[4m, tbc=\E[3g,
    u6=\E[%i%d;%dR, u7=\E[6n, u8=\E[?1;2c, u9=\E[c,
    vpa=\E[%i%p1%dd,

iTerm beta (corruption occurs)

#   Reconstructed via infocmp from file: /Users/q/Applications/iTerm.app/Contents/Resources/terminfo/78/xterm-256color
xterm-256color|xterm with 256 colors,
    am, bce, ccc, km, mc5i, mir, msgr, npc, xenl,
    colors#256, cols#80, it#8, lines#24, pairs#32767,
    acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
    bel=^G, blink=\E[5m, bold=\E[1m, cbt=\E[Z, civis=\E[?25l,
    clear=\E[H\E[2J, cnorm=\E[?12l\E[?25h, cr=^M,
    csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
    cud=\E[%p1%dB, cud1=^J, cuf=\E[%p1%dC, cuf1=\E[C,
    cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,
    cvvis=\E[?12;25h, dch=\E[%p1%dP, dch1=\E[P, dim=\E[2m,
    dl=\E[%p1%dM, dl1=\E[M, ech=\E[%p1%dX, ed=\E[J, el=\E[K,
    el1=\E[1K, flash=\E[?5h$<100/>\E[?5l, home=\E[H,
    hpa=\E[%i%p1%dG, ht=^I, hts=\EH, ich=\E[%p1%d@,
    il=\E[%p1%dL, il1=\E[L, ind=^J, indn=\E[%p1%dS,
    initc=\E]4;%p1%d;rgb\:%p2%{255}%*%{1000}%/%2.2X/%p3%{255}%*%{1000}%/%2.2X/%p4%{255}%*%{1000}%/%2.2X\E\\,
    invis=\E[8m, is2=\E[!p\E[?3;4l\E[4l\E>, kDC=\E[3;2~,
    kEND=\E[1;2F, kHOM=\E[1;2H, kIC=\E[2;2~, kLFT=\E[1;2D,
    kNXT=\E[6;2~, kPRV=\E[5;2~, kRIT=\E[1;2C, ka1=\EOw,
    ka3=\EOy, kb2=\EOu, kbs=^H, kc1=\EOq, kc3=\EOs, kcbt=\E[Z,
    kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA,
    kdch1=\E[3~, kend=\EOF, kent=\EOM, kf1=\EOP, kf10=\E[21~,
    kf11=\E[23~, kf12=\E[24~, kf13=\E[1;2P, kf14=\E[1;2Q,
    kf15=\E[1;2R, kf16=\E[1;2S, kf17=\E[15;2~, kf18=\E[17;2~,
    kf19=\E[18;2~, kf2=\EOQ, kf20=\E[19;2~, kf21=\E[20;2~,
    kf22=\E[21;2~, kf23=\E[23;2~, kf24=\E[24;2~,
    kf25=\E[1;5P, kf26=\E[1;5Q, kf27=\E[1;5R, kf28=\E[1;5S,
    kf29=\E[15;5~, kf3=\EOR, kf30=\E[17;5~, kf31=\E[18;5~,
    kf32=\E[19;5~, kf33=\E[20;5~, kf34=\E[21;5~,
    kf35=\E[23;5~, kf36=\E[24;5~, kf37=\E[1;6P, kf38=\E[1;6Q,
    kf39=\E[1;6R, kf4=\EOS, kf40=\E[1;6S, kf41=\E[15;6~,
    kf42=\E[17;6~, kf43=\E[18;6~, kf44=\E[19;6~,
    kf45=\E[20;6~, kf46=\E[21;6~, kf47=\E[23;6~,
    kf48=\E[24;6~, kf49=\E[1;3P, kf5=\E[15~, kf50=\E[1;3Q,
    kf51=\E[1;3R, kf52=\E[1;3S, kf53=\E[15;3~, kf54=\E[17;3~,
    kf55=\E[18;3~, kf56=\E[19;3~, kf57=\E[20;3~,
    kf58=\E[21;3~, kf59=\E[23;3~, kf6=\E[17~, kf60=\E[24;3~,
    kf61=\E[1;4P, kf62=\E[1;4Q, kf63=\E[1;4R, kf7=\E[18~,
    kf8=\E[19~, kf9=\E[20~, khome=\EOH, kich1=\E[2~,
    kind=\E[1;2B, kmous=\E[M, knp=\E[6~, kpp=\E[5~,
    kri=\E[1;2A, mc0=\E[i, mc4=\E[4i, mc5=\E[5i, meml=\El,
    memu=\Em, mgc=\E[?69la, oc=\E]104\007, op=\E[39;49m,
    rc=\E8, rep=%p1%c\E[%p2%{1}%-%db, rev=\E[7m, ri=\EM,
    rin=\E[%p1%dT, ritm=\E[23m, rmacs=\E(B, rmam=\E[?7l,
    rmcup=\E[?1049l\E[23;0;0t, rmir=\E[4l, rmkx=\E[?1l\E>,
    rmm=\E[?1034l, rmso=\E[27m, rmul=\E[24m,
    rs1=\Ec\E]104\007, rs2=\E[!p\E[?3;4l\E[4l\E>, sc=\E7,
    setab=\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48\:5\:%p1%d%;m,
    setaf=\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38\:5\:%p1%d%;m,
    setb=\E[4%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m,
    setf=\E[3%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m,
    sgr=%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p5%t;2%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m,
    sgr0=\E(B\E[m, sitm=\E[3m, smacs=\E(0, smam=\E[?7h,
    smcup=\E[?1049h\E[22;0;0t, smir=\E[4h, smkx=\E[?1h\E=,
    smm=\E[?1034h, smso=\E[7m, smul=\E[4m, tbc=\E[3g,
    u6=\E[%i%d;%dR, u7=\E[6n, u8=\E[?%[;0123456789]c,
    u9=\E[c, vpa=\E[%i%p1%dd,
edgar-bonet commented 9 months ago

@quinncomendant: Interesting! You have two terminal description files named xterm-256color:

These files are different: some escape sequences differ, and the second file declares some terminal capabilities that are not in the first one. It would be interesting to see whether the problem is related to those differences.

In iTerm beta, there must be some environment variable providing a non-standard search path for the terminfo database. You may look for $TERMINFO or $TERMINFO_DIRS. If neither is defined, you may try:

env | grep terminfo

Could you try to unset this variable in order to run iTerm beta with the standard, system-provided, terminfo database? You may also set the variable in iTerm stable in order to run it with the database from iTerm beta. This should tell us whether the issue is related to this terminal description.

quinncomendant commented 9 months ago

@edgar-bonet Indeed, the iTerm beta terminfo is set via a custom TERMINFO_DIRS variable. Setting this to TERMINFO_DIRS=/usr/share/terminfo allows iTerm to use the macOS standard /usr/share/terminfo/78/xterm-256color file, and the corruption doesn't happen.

So, it seems like we're narrowing down the necessary conditions to: ttyplot at commit c73270c + iTerm beta terminfo file (which is on GitHub, if you'd like to test this presumption on your computer).

edgar-bonet commented 9 months ago

@quinncomendant: I ran your test (for i in 13.669 ...) on both ttyplot c73270c and master (035ff43), using the xterm-256color terminal description from iTerm beta you linked to. I checked with strace that it is indeed using that terminal description file. In both cases, the display on my computer had no glitches.

I am using:

Although I cannot assert it's iTerm's fault (or its terminfo file), I would suggest you file an issue on their bug tracker. iTerm's developers have probably much more expertise on terminal description files than anyone here.

It would be nice to know whether that gets fixed on their side.

quinncomendant commented 9 months ago

I had already done so.