tmux unicode issues in the Alpine docker container

[VorpalBlade]

This is the (possible?) tmux issue I mentioned in issue #146. I'm not convinced it is an actual bug, as I can only reproduce it in the Alpine docker image.

First issue

Steps to reproduce

Host terminal is konsole on Ubuntu 20.04 (and mate-terminal on Arch, tried it with both). TERM=xterm-256color in both cases. In both cases the font used is "Terminus (TTF) 12pt"

1. Start docker image as described in README

   docker run -e TERM -e COLORTERM -w /root -it --rm alpine sh -uec '
   apk add zsh curl tmux
   sh -c "$(curl -fsSL https://raw.githubusercontent.com/romkatv/zsh4humans/v5/install)"'

2. Select 2 (PC), 1 (Standard keys). (Not sure if this matters).
3. Select "n" to if you use tmux. This will result in getting the integrated tmux option.
4. Select y (diamond looks rotated square), n (lock), n (lock).
5. Select 2 (classic), 1 (unicode), 3 (dark), 1 (no), 1 (angled), 1 (sharp), 1 (flat), 1 (one line), 1 (compact), 1 (concise), n (no).
6. echo 'set -g default-terminal "tmux-256color"' > .tmux.conf (otherwise p10k will want to reconfigure, but the issue occurs even without that)
7. Start a nested tmux. Notice prompt look wrong (angled unicode characters missing).

Broken appearance (nested tmux):

Non-nested (correct look):

A way to fix this in the Alpine image is exporting LANG as a UTF-8 locale, e.g. LANG=en_US.UTF-8. This makes me suspect that it might be an issue with the Alpine image rather than with z4h. So maybe all that is needed is an update to the example for Alpine in the README?

The Ubuntu image doesn't have this problem, as it seems like by default LC_ALL=C.UTF-8 is set. (I don't understand why they set LC_ALL rather than LANG though in the image. LC_ALL overrides everything, while LANG will be the default that can be overridden, see output of locale as well as man 7 locale if you are interested in the priority order between the various locale environment variables.)

Second related issue

In addition I found another strange issue while investigating this issue. It happens both in the Alpine and the Ubuntu containers: If I select "y" to the prompt that I use tmux, I get much fewer options for the prompt when customising p10k prompt. I also notice that $TERM is different inside "true" tmux and "integrated" tmux, which is likely the cause of this. Setting a 256-color TERM in .tmux.conf fixes the issue. Might not be a z4h bug, it is just surprising that the integrated tmux behaves differently than standard tmux here.

[romkatv]

Thanks for the detailed description. Great stuff. It allows me to address all problems in one go.

5. echo 'set -g default-terminal "tmux-256color"' > .tmux.conf (otherwise p10k will want to reconfigure, but the issue occurs even without that)

If you run tmux without ~/.tmux.conf, tmux will create a TTY that supports only 8 colors. When zsh4humans initializes in this TTY, it sees that ~/.p10k.zsh that you've generated previously won't work because it requires 256 colors. So zsh4humans asks you to create a new prompt config that it'll store separately. The next time you end up in a TTY with 16 colors, you won't need to go through the configuration wizard because you'll already have a config for just such an occasion. If you run zsh4humans from a dumb linux terminal, you might once again see configuration wizard if none of your existing configs work in pure ASCII mode. You can see my p10k configs here: https://github.com/romkatv/dotfiles-public. I have two files with content: .p10k.zsh for terminals with all capabilities and .p10k-ascii-8color.zsh for gimped terminals. The other two configs use in-between capabilities and I decided to simply fall back to p10k-ascii-8color.zsh there. zsh4humans looks at files with these names when it decides which p10k config to load.

Note: You can run echoti colors to check how many colors the current TTY supports.

6. Start a nested tmux. Notice prompt look wrong (angled unicode characters missing). Broken appearance (nested tmux):

Let's see what we get if we run tmux in the same alpine image but without zsh4humans.

❯ docker run -e TERM -e COLORTERM -w /root -it --rm alpine sh
~ # apk -q add zsh tmux
~ # zsh
04b79946c712# print '\u276F'
❯
04b79946c712# SHELL=/bin/zsh tmux
04b79946c712:~# print '\u276F'
_
04b79946c712:~# exit
[exited]
04b79946c712# SHELL=/bin/zsh tmux
04b79946c712:~# print '\u276F'
❯

I've executed print '\u276F' three times.

1. In plain zsh. It worked.
2. In zsh under tmux. It printed _ instead of ❯.
3. In zsh under tmux -u. It worked.

This is how tmux works. It's by design. From man tmux:

-u   Write UTF-8 output to the terminal even if the first environment
     variable of LC_ALL, LC_CTYPE, or LANG that is set does not contain
     "UTF-8" or "UTF8".

On alpine neither of these environment variables is set, so tmux replaces all characters outside of ASCII range with _ unless you invoke it with -u.

So maybe all that is needed is an update to the example for Alpine in the README?

Spot on! I've added LC_ALL=C to the docker invocation in the docs to avoid issues of this kind.

In addition I found another strange issue while investigating this issue. It happens both in the Alpine and the Ubuntu containers: If I select "y" to the prompt that I use tmux, I get much fewer options for the prompt when customising p10k prompt.

When you say that you use tmux, zsh4humans will start tmux automatically. It's the real tmux, the same thing you would get if you run tmux command manually. As such, it respects ~/.tmux.conf. If there is no ~/.tmux.conf, tmux wil use defaults. The default value of TERM inside tmux is screen, which supports only 8 colors. When powerlevel10k asks you about your preferences w.r.t. prompt, it won't show options that your TTY cannot render. Hence no options that require more than 8 colors.

People who actually use tmux usually have ~/.tmux.conf and they usually enable a more powerful TERM. The standard choices are screen-256color and tmux-256color.

I also notice that $TERM is different inside "true" tmux and "integrated" tmux. It is just surprising that the integrated tmux behaves differently than standard tmux here.

TERM in "true" tmux is what you explicitly request in ~/.tmux.conf. The "integrated" tmux is not tmux at all. It doesn't have any features of tmux, doesn't read ~/.tmux.conf, doesn't respond to Ctrl-B and doesn't export TMUX environment variable. It tries to be as invisible as possible and to preserve all capabilities of the parent terminal. Currently it uses tmux-256color as TERM but this can change in the future.

[romkatv]

The "integrated" tmux is not tmux at all.

I've updated the installer to avoid this confusion with "integrated" tmux. Now, instead of "do you use tmux?" the installer asks "do you want zsh to always run in tmux?". Users uncorrupted by prior exposure to zsh4humans will naturally understand "tmux" to mean the real tmux. If you reply "yes" to the question, you get these lines in zshrc:

# Start tmux if not already in tmux.
zstyle ':z4h:' start-tmux       command tmux -u new -A -D -t z4h

From the looks of this line it should be clear that the exact tmux command can be changed.

If you reply "no" to the question about tmux, you don't get anything in zshrc that mentions tmux.

What do you think?

[romkatv]

If I select "y" to the prompt that I use tmux, I get much fewer options for the prompt when customising p10k prompt.

I've added another paragraph (the second paragraph on the screenshot below) to the welcome message that gets printed in this case. It should help at least those users who read what gets printed. I'm afraid it's a small fraction but I've done what I could.

I don't intend to do anything else on this issue. Please let me know if I've missed anything.

[VorpalBlade]

I can't think of anything else you could or should reasonably do. I'm am however personally curious as to the difference (if any) between screen-256color and tmux-256color. Is there a reason to prefer one over the other?

[romkatv]

The primary difference is that tmux-256color supports 24-bit colors. This terminfo is much newer than screen-256colors and thus not available on all machines. If you enable ssh teleportation in z4h, this won't be a problem because z4h will teleport terminfo, too. But if you do bare-bone ssh, you can run into trouble when using tmux-256color.

If you haven't tried ssh teleportation yet, give it a shot. It's a great feature. Search for "ssh" in ~/.zshrc and read comments.

I'm closing this issue as fixed. Thanks for filing it!

[VorpalBlade]

I did try ssh teleportation. I'm not that keen on it, as it potentially overwrites local config files (.zshenv, .zshrc etc). It would be better if it put the teleported files in a subdirectory instead. I manage my config files using chezmoi (something I need to figure out how to combine with z4h as well once I switch to it fully).

Also I saw something about copying the history back (maybe in the mega issue?). This does not seem useful at all to me. I generally execute very different commands on a remote server, a research robot and on my laptop. Is it possible to turn that off?

[romkatv]

I did try ssh teleportation. I'm not that keen on it, as it potentially overwrites local config files (.zshenv, .zshrc etc). It would be better if it put the teleported files in a subdirectory instead. I manage my config files using chezmoi (something I need to figure out how to combine with z4h as well once I switch to it fully).

I'm aware of chezmoi and other similar approaches. I think what zsh4humans does is better. It allows you to have all dotfiles on a local dev machines and they just follow you around. This way you won't have out-of-sync dotfiles and you no longer need to install anything related to shell on remove machines (I don't install zsh on prod machines because I usually don't need it; if and when I ssh to a machine, zsh will get installed automatically by z4h). You can easily have custom configuration for different hosts in dotfiles on your local machine (you already do in your dotfiles, right?) either split into different files or within the same file under if. And you can configure which files to send to which machine when you connect to it. No pressure though, if chezmoi works for you, then it works. zsh4humans is useful even if you use the surface part of it.

Also I saw something about copying the history back (maybe in the mega issue?). This does not seem useful at all to me. I generally execute very different commands on a remote server, a research robot and on my laptop.

As do I. I run different commands on a dev machine, on network routers, on compute cluster machines, etc. Whenever I ssh to a router, its history contains the commands from that specific router, then commands from all other routers, then commands from all machines with the same OS. If I wipe a router and rebuild it from an image, it'll still have the same command history when I ssh to it next time.

Pulling history to a local machine from a remote is just that -- pulling a history file. You decide which history files to send where and which history files to load independently from that. Pulling files to a local machine allows you to do all of those things and more (e.g., you can back them up) but it doesn't force you to do any of it.

Is it possible to turn that off?

Naturally. Although it might be simpler to simply not enable it 😁 It's off by default.

[romkatv]

I need to figure out how to combine [chezmoi ] with z4h as well once I switch to it fully

You need to add .zshrc, .zshenv and .p10k.zsh to your dotfiles repo. That's it. If you don't enable ssh teleportation, there is nothing else you need to do.

I also just realized that we might have been talking about different things. I use a similar tool to chezmoi to synchronize my dotfiles across dev machines. These are the machines on which I have ssh keys and on which I physically bang on the keyboard. I've a few laptops and a desktop. When I ssh between the dev machines, I don't use ssh teleportation but for all other machines I do. That is, I enable ssh teleportation by default and blacklist it for a handful of machines. You can try something like that or start more conservatively by enabling ssh teleportation for a whitelisted set of machines. Or don't enable it at all -- it's also fine.

[VorpalBlade]

You need to add .zshrc, .zshenv and .p10k.zsh to your dotfiles repo. That's it. If you don't enable ssh teleportation, there is nothing else you need to do.

Good to know.

I also just realized that we might have been talking about different things. Yes indeed. Then it starts to make a little bit more sense. I was just about to write about that same issue.

Related however: Is there a way for ssh teleportation to teleport additional stuff, say tmux/gdb/git configs or extra binaries to machines where I cannot install software as root (such as on the GPU server I have access to at university)?

[romkatv]

You need to add .zshrc, .zshenv and .p10k.zsh to your dotfiles repo. That's it. If you don't enable ssh teleportation, there is nothing else you need to do.

Good to know.

Just wanted to clarify in case it wasn't clear (I realize that z4h isn't clear at all, so the fault is mine): the installer simply creates these files. It doesn't do anything else. So if you save these files, you don't need to run the installer again.

Related however: Is there a way for ssh teleportation to teleport additional stuff, say tmux/gdb/git configs or extra binaries to machines where I cannot install software as root (such as on the GPU server I have access to at university)?

Yep. Search for send-extra-files in ~/.zshrc.

It's a good idea to read the whole ~/.zshrc that you've got. There are things in it that it tells you to delete.