Man page colorization issues with aws-cli

[exploide]

I like to use bat for colorization of man pages and have the following environment variables set:

MANPAGER="sh -c 'col -bx | bat -l man -p'"
MANROFFOPT="-c"

This works great for the regular man pages on my system.

Now the aws-cli tool has a help command that also opens a man pager to display the manual. But the escapes are wrong there.

What steps will reproduce the bug?

1. install bat and set the aforementioned environment variables
2. install aws-cli
3. aws s3 help

What happens?

See the 1m, 0m, 22m escapes in the screenshot.

This works correctly if I unset the environment variables but then there is no color.

What did you expect to happen instead?

I expected the help page to be colorized like the regular man pages.

But I'm not sure whether this is a problem in bat, aws, or my configuration.

How did you install bat?

• bat 0.18.3 from package management in Fedora 36
• aws-cli/2.7.6

[keith-hall]

Thanks for reporting. I wasn't able to reproduce this:

python3 -m pip install --user awscli
Collecting awscli
  Downloading awscli-1.25.41-py3-none-any.whl (3.9 MB)
     |████████████████████████████████| 3.9 MB
Collecting docutils<0.17,>=0.10
  Downloading docutils-0.16-py2.py3-none-any.whl (548 kB)
     |████████████████████████████████| 548 kB 
Collecting botocore==1.27.41
  Downloading botocore-1.27.41-py3-none-any.whl (9.0 MB)
     |████████████████████████████████| 9.0 MB
Requirement already satisfied: colorama<0.4.5,>=0.2.5 in /usr/lib/python3/dist-packages (from awscli) (0.4.3)
Requirement already satisfied: PyYAML<5.5,>=3.10 in /usr/lib/python3/dist-packages (from awscli) (5.3.1)
Collecting rsa<4.8,>=3.1.2
  Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting s3transfer<0.7.0,>=0.6.0
  Downloading s3transfer-0.6.0-py3-none-any.whl (79 kB)
     |████████████████████████████████| 79 kB
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore==1.27.41->awscli) (1.25.8)
Collecting python-dateutil<3.0.0,>=2.1
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |████████████████████████████████| 247 kB
Collecting jmespath<2.0.0,>=0.7.1
  Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting pyasn1>=0.1.3
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.27.41->awscli) (1.14.0)
Installing collected packages: docutils, python-dateutil, jmespath, botocore, pyasn1, rsa, s3transfer, awscli
Successfully installed awscli-1.25.41 botocore-1.27.41 docutils-0.16 jmespath-1.0.1 pyasn1-0.4.8 python-dateutil-2.8.2 rsa-4.7.2 s3transfer-0.6.0

MANPAGER="sh -c 'col -bx | /home/keith/repos/bat/target/release/bat -l man -p'" MANROFFOPT="-c" aws s3 help

It may be worth trying to upgrade to bat 0.20 and see if that fixes it. Though I note that the AWS CLI version which got installed for me seems vastly different to what you reported. I'll try again with v2 when I get time.

[exploide]

I tried again using bat 0.21.0 and both, aws-cli 1.x and 2.x and the issue is still present.

Interesting that it isn't reproducible for you. Maybe it has something to do with which groff/roff is used? I am on a Fedora system here and could provide further details if I knew what is of interest.

[keith-hall]

I have tried with aws-cli/2.7.20 and bat 0.21 on my system and it's also fine for me.

groff --version

GNU groff version 1.22.4 Copyright (C) 2018 Free Software Foundation, Inc. GNU groff comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of groff and its subprograms under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING.

called subprograms:

GNU troff (groff) version 1.22.4 GNU grops (groff) version 1.22.4

FWIW I'm using Linux Mint 20.3. I'm not sure how else to debug this tbh and thus what else would be of interest, I'm hoping one of the other maintainers will have ideas :)

[exploide]

Ok, thanks for your effort anyway!

I spun up fresh VMs, one with Fedora 36, one with Linux Mint 21 and performed the exact same steps afterwards:

wget https://github.com/sharkdp/bat/releases/download/v0.21.0/bat-v0.21.0-x86_64-unknown-linux-gnu.tar.gz
tar xzf bat-v0.21.0-x86_64-unknown-linux-gnu.tar.gz

export PATH=./bat-v0.21.0-x86_64-unknown-linux-gnu:$PATH
export MANPAGER="sh -c 'col -bx | bat -l man -p'"
export MANROFFOPT="-c"

wget https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip
unzip awscli-exe-linux-x86_64.zip
sudo aws/install

man grep
aws s3 help

And you are right, on Linux Mint, it is not reproducible, but on Fedora it is. While man grep looks nice in both cases, aws s3 help is only fine on Mint.

So maybe there is a difference with the man pager stuff between Mint and Fedora. But I cannot spot a difference between the installed man and groff.

[ryaminal]

So, I'm a noob. but i decided to look around the aws cli code. Here is a line that calls groff -m man -T ascii. perhaps this is a clue?

And for anyone curious, here is a Containerfile that can be used to reproduce:

FROM amazon/aws-cli

# install deps(util-linux has `col`)
RUN yum install -y tar gzip util-linux

# download and extract bat
RUN curl -OL https://github.com/sharkdp/bat/releases/download/v0.22.1/bat-v0.22.1-x86_64-unknown-linux-gnu.tar.gz \
  && tar xvf bat-v0.22.1-x86_64-unknown-linux-gnu.tar.gz \
  && mv bat-v0.22.1-x86_64-unknown-linux-gnu/bat /usr/local/bin \
  ;

# set pager envs
ENV MANPAGER="sh -c 'col -bx | bat -l man -p'"
ENV MANROFFOPT="-c"
ENV BAT_THEME='Solarized (dark)'

and then:

podman build . -t aws-cli-bat
podman run -it --rm aws-cli-bat help

it appears that aws doesn't respect the MANROFFOPT env flag.

Also. found out that if the col -bx section is removed from the MANPAGER env, then most of the problems are fixed. still some problems which can be seen in aws help and looking at the types for some options I see --debug [0m22m(boolean)

[tobywan]

found this issue as also experiencing problem:

macos 13.0.1 (22A400) (arm)
❯ fish --version
fish, version 3.5.1
❯ alacritty --version
alacritty 0.11.0 (8dbaa0b)
❯ bat --version
bat 0.22.1
❯ aws --version
aws-cli/2.9.1 Python/3.11.0 Darwin/22.1.0 source/arm64 prompt/off
❯ groff --version
GNU groff version 1.22.4
Copyright (C) 2018 Free Software Foundation, Inc.
GNU groff comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of groff and its subprograms
under the terms of the GNU General Public License.
For more information about these matters, see the file
named COPYING.

called subprograms:

GNU grops (groff) version 1.22.4
GNU troff (groff) version 1.22.4

As per ryaminal's workaround above, setting

export MANPAGER="sh -c 'bat -l man -p'"

worked for me for the titles, but not the "inline" colorizations:

[ValentinLeTallec]

I did not try specifically with aws, but I had a similar problem with other man pages, a workaround was to remove the original colors (the sed ) and then let bat handle them instead :

export MANPAGER="sh -c 'sed -r \"s/\x1B\[([0-9]{1,3}(;[0-9]{1,2};?)?)?[mGK]//g\" | bat --language man --plain'"

[Moulick]

Syntax highlighting does not work for me at all for aws-cli, I get a simple white output 😢. I have the following config

aws-cli/2.15.14 Python/3.11.7 Darwin/23.3.0 source/arm64 prompt/off bat 0.24.0

alias cat="bat"
export MANPAGER="sh -c 'col -bx | bat -l man -p'"
export MANROFFOPT="-c"

alias bathelp='bat --plain --language=help'
help() {
    "$@" --help 2>&1 | bathelp
}

alias -g -- -h='-h 2>&1 | bat --language=help --style=plain'
alias -g -- --help='--help 2>&1 | bat --language=help --style=plain'