corrupt graph display, apparently unhandled terminal control sequences displayed

[OSS542]

I get a corrupt "graph" column display. There are also what appear to be unhandled terminal control sequences displayed at the end of the "title" line, and in front of some of the command names. The xterm is with TERM=xterm-256color. and the curses library is ncurses-6.4

[OSS542]

The "u" ascii mode eliminates almost all of this.

[bbonev]

Sounds like something is not configured. Can you also share the content of LANG, LC_* environment variables, the version of xterm and OS used? That can help to try to reproduce it.

I did just try with xterm 366, ncurses 6.2, LANG=C.UTF-8, no LC_* set, TERM=xterm-256-color and it works as expected. Via the same xterm over ssh to a system with ncurses 6.4 with Debian patches also seems to work as expected.

[OSS542]

I am wondering which unicode fonts you are using with this. The unicode functionality seems ok on xterm here, but I suspect I may be missing a unicode font with the necessary characters.

Linux DellXPS 6.1.0 #4 SMP Thu Jan 19 21:17:27 JST 2023 x86_64 GNU/Linux custom system built entirely from source code

using ncurses-6.4

bash-5.1$ xterm -version XTerm(359)

LANG=en_US.utf-8

output from locale: LANG=en_US.utf-8 LC_CTYPE="en_US.utf-8" LC_NUMERIC="en_US.utf-8" LC_TIME="en_US.utf-8" LC_COLLATE="en_US.utf-8" LC_MONETARY="en_US.utf-8" LC_MESSAGES="en_US.utf-8" LC_PAPER="en_US.utf-8" LC_NAME="en_US.utf-8" LC_ADDRESS="en_US.utf-8" LC_TELEPHONE="en_US.utf-8" LC_MEASUREMENT="en_US.utf-8" LC_IDENTIFICATION="en_US.utf-8" LC_ALL=

[bbonev]

Linux DellXPS 6.1.0 https://github.com/Tomas-M/iotop/pull/4 SMP Thu Jan 19 21:17:27 JST 2023 x86_64 GNU/Linux custom system built entirely from source code

Cool :)

The settings above look sane. Also I doubt that the problem is related to the newer versions of the tools/libs used.

My desktop/laptop(s) are all Devuan with mostly untouched settings and no additional terminal related fonts installed. By default xterm uses fixed font. In my preferred terminal the font is Monospace (fonts-noto-mono 20201225-1).

Note that this may also depend on fontconfig where certain Unicode parts may be substituted and come from a different font than the one in use, thus adding another font may fix it for the ones you already have.

[OSS542]

found the package fonts-noto-mono 20201225-1 for debian, but didn't see anything called Monospace. I experimented with various fonts in there, but no luck with the graph display.

[OSS542]

Is it possible to have iotop start up in ascii mode ?

[bbonev]

Looks like I was confused about the fonts and have misled you...

$ fc-match monospace
VeraMono.ttf: "Bitstream Vera Sans Mono" "Roman"

Also with xterm it seems important how you start it, e.g. LANG=C xterm -fa monospace never works for me, regardless how LANG is set from inside the terminal afterwards. When LANG is set to e.g. C.UTF-8 or en_US.UTF-8 then it works.

Currently there is no way to start directly in non-UTF mode, I will add an option for preferring ASCII on start.

[OSS542]

Thank you very much for the ascii mode modification...:-)

I was able to install the "Bitstream Vera Sans Mono" font (VeraMono.ttf), but no luck with the graph display from that. Xterm is opening the proper font files according to strace, so it looks like a different font is actually needed.

from strace: /usr/share/fonts/ttf-bitstream-vera-1.10/VeraMoBd.ttf /usr/share/fonts/ttf-bitstream-vera-1.10/VeraMoBI.ttf /usr/share/fonts/ttf-bitstream-vera-1.10/VeraMoIt.ttf /usr/share/fonts/ttf-bitstream-vera-1.10/VeraMono.ttf

[OSS542]

just a thought if I may: Could you run xterm under strace with log output to a file, then run iotop in that xterm ? Grepping the log for the fonts that are opened should show what actually gets used. An example: strace -o xyzzy.log -f -y -s 256 xterm # run iotop in this xterm with graph then exit

[bbonev]

Did you verify that xterm is at all capable of displaying Unicode? Try this:

echo -e "\xd0\x90\xd0\x92\xd0\xa1"

should produce

АВС

These are Cyrillic letters, visually same as ABC.

Graphs are constructed from Braille characters, here is an example how to reproduce with simple echo. That is why I think an strace will not show anything useful, because the graphs are just like the below.

echo "⢀⣦⠀⣸⡇⢀⣾⣀⢰⣀"

or the same encoded (in case copying mangles the above)

echo -e "\xe2\xa2\x80\xe2\xa3\xa6\xe2\xa0\x80\xe2\xa3\xb8\xe2\xa1\x87\xe2\xa2\x80\xe2\xa3\xbe\xe2\xa3\x80\xe2\xa2\xb0\xe2\xa3\x80"

I am more inclined to believe that this is not a font issue but xterm is not started properly and Unicode support is disabled, see option +u8. As I said 3 messages above, when xterm is started with a non-Unicode locale, setting LANG from inside will not change anything and it will not properly display Unicode. I believe that I could reproduce your problem in that way. There are two possible things to do - start xterm with a locale that will hint Unicode support or modify the settings at runtime.

Start a new one like this:

LANG=en_US.UTF-8 TERM=xterm xterm

and then see if Unicode works inside.

To modify the settings at runtime - Ctrl-Right Click (and hold) will open a pop-up menu and there is a UTF-8 Encoding option... After changing that option, use Ctrl-L to redraw iotop or just restart it.

While I have no problem to help you with debugging this issue, I am sure that it is not related to iotop and I would prefer to keep it closed.

[OSS542]

Finally found the issue.....It was a problem with the entity between the keyboard and the chair..... The "echo" command line you sent me was a great help with this. It turns out that I had linked with ncurses instead of ncursesw (wide library). I did not know about the ncursesw library being used with a unicode locale. This meant that ordinary characters would display well, but certain less commonly used characters would not. This made it rather challenging to find the problem....:-)
Your kind assistance and attention is very much appreciated. Many Thanks ....:-)

[bbonev]

Ah, wouldn't have guessed that! BTW. Can you contribute build instructions for LFS or whatever else you are using? That would help others not to repeat the same mistake. PS. Maybe I can also add a run or compile time check and disable unicode in case ncurses does not have it...

[OSS542]

Some linux distributions may not have libbsd installed, but it is required for iotop-c. One must also be careful to link to ncursesw, and not ncurses or curses if one is using unicode. The ncursesw library is source compatible with ncurses, but not binary compatible. I've attached a patch to iotop-1.22 which addresses this. I think it may be useful. iotop-1.22.patch

[OSS542]

The following should now work without issue on linux as in your installation notes : sudo make install

[bbonev]

I don't understand the reasons behind the proposed patch. Nothing in iotop depends on or requires libbsd.

As long as iotop is Linux specific, the development is on glibc with compatibility for oldish glibc, musl and uClibc. On the other hand, the cost of adding extra dependencies or conditional code will be very high in the future, so I would prefer to avoid both whenever feasible.

[OSS542]

That patch was what I had to do to get iotop to build. Some of those BSD type "realloc" functions did not exist in any of the shared link libraries I already had, including those from glibc. I had to install libbsd to get them. My glibc is an older one, so that may be why. I do not know if the situation has since changed. I should probably update glibc at some point. In any case, it builds well, and works nicely now that I've used ncursesw instead of ncurses.

[bbonev]

realloc is in glibc since the very beginning - it is not possible to be missing. Header file is <stdlib.h> and library is libc (linker option -lc which should be assumed by default a sane compiler).

The BSD extension that is only present in recent glibc versions is reallocarray but that is not used by iotop and should not be relevant in any way.

How old are your glibc/gcc? Sounds like something is very broken with gcc and/or glibc.

By default iotop uses -flto and that does not work well with very old gcc.

Things to try:

• rebuild without flto: make re V=1 NO_FLTO=1
• rebuild with explicitly asking to link libc: make re V=1 LIBS=-lc
• both of the above: make re NO_FLTO=1 V=1 LIBS=-lc
• rebuild without any options: make re V=1 NO_FLTO=1 MYCFLAGS=--std=gnu90 MYLDFLAGS="-lncursesw -ltinfo"
• same with libc: make re V=1 NO_FLTO=1 MYCFLAGS=--std=gnu90 MYLDFLAGS="-lncursesw -ltinfo -lc"

Play with various combinations of CFLAGS to find what works. Note that --std=gnu90 is essential and can not be omitted.

I have tried with gcc 4.4.7, glibc 2.13 and ncurses 5.9 on kernel 3.2.96 (the most ancient VM I still keep) and everything works by removing the unsupported options.

[OSS542]

Interesting.....I knew realloc was present from the beginning, but I hadn't heard that the bsd extensions had been made available. The glibc I have presently is 2.19, which is quite old (around 2014 I think). The gcc is 10.2.0, which I think is from 2020. Given what you've told me, I think my glibc overdue for replacement. I'll also pull the libbsd back out. I do enjoy building from source, so it will keep me busy for a while. Many thanks again.... I'll stop bothering you...:-)