CLI UTF-8 decomposer for text analysis capable of displaying Unicode code point names and categories, along with ASCII control characters, UTF-16 surrogate pair pieces, invalid UTF-8 sequences parts as separate bytes, etc.
A necessity for a tool that can quickly identify otherwise indistinguishable Unicode code points.
pipx
(recommended)pipx install holms
curl -sS https://github.com/es7s/holms/blob/master/install.sh | sh
Usage: holms run [OPTIONS] [INPUT]
Read data from INPUT file, find all valid UTF-8 byte sequences, decode them and display as
separate Unicode code points. Use '-' as INPUT to read from stdin instead.
The application works in two modes: buffered (the default if INPUT is a
file) and unbuffered (default when reading from stdin). Options -b
/-u
explicitly override output mode regardless of the default setting.
In buffered mode the result begins to appear only after EOF is encountered (i.e., the WHOLE file has been read to the buffer). This is suitable for short and predictable inputs and produces the most compact output with fixed column sizes.
The unbuffered mode comes in handy when input is an endless piped stream: the results will be displayed in real time, as soon as the type of each byte sequence is determined, but the output column widths are not fixed and can vary as the process goes further.
Despite the name, the app actually uses tiny (4 bytes) input buffer, but it's the only way to handle UTF-8 stream and distinguish valid sequences from broken ones; in truly unbuffered mode the output would consist of ASCII-7 characters (
0x00
-0x7F
) and unrecognized binary data (0x80
-0xFF
) only, which is not something the application was made for.
Options:
-b, --buffered / -u, --unbuffered
Explicitly set to wait for EOF before processing the
output (buffered), or to stream the results in
parallel with reading, as soon as possible
(unbuffered). See BUFFERING section above for the
details.
-m, --merge Replace all sequences of repeating characters with one
of each, together with initial length of the sequence.
-g, --group Group the input by code points (=count unique), sort
descending and display counts instead of normal
output. Implies '--merge' and forces buffered ('-b')
mode. Specifying the option twice ('-gg') results in
grouping by code point category instead, while doing
it thrice ('-ggg') makes the app group the input by
super categories.
-f, --format Comma-separated list of columns to show (order is
preserved). Run 'holms format' to see the details.
-n, --names Display names instead of abbreviations. Affects `cat`
and `block` columns, but only if column in question is
already present on the screen. Note that these columns
can still display only the beginning of the attribute,
unless '-r' is provided.
-a, --all Display ALL columns.
-r, --rigid By default some columns can be compressed beyond the
nominal width, if all current values fit and there is
still space left. This option disables column
shrinking (but they still will be expanded when
needed).
--decimal Use decimal byte offsets instead of hexadecimal.
--alt Use alternative notation for control characters: caret
notation for ASCII C0, octal notation for ASCII C1.
--oneline Discard all newline characters (0x0a LINE FEED) from
the input.
--no-table Do not format results as a table, just apply the
colors to characters (equivalent to '-f char', implies
'-b'). Compatible with '-merge', '--format' and even '
--group'.
--no-override Do not replace control/whitespace code point markers
with distinguishable characters ('▯' to '↵', '␣' etc).
Run 'holms legend' to see the details.
-?, --help Show this message and exit.
Option -f
/--filter
can be used to specify what columns to display. As an
alternative, there is an -a
/--all
option that enables displaying of all
currently available columns.
Also -m
/--merge
option is demonstrated, which tells the app to collapse
repetitive characters into one line of the output while counting them:
There is an official Unicode Consortium data file included in the repository for
test purposes, named confusables.txt. In the next
example we extract line #3620 using sed
, delete all TAB (0x08
) characters
and feed the result to the application. The result demonstrates various Unicode
dot/bullet code points:
-g
/--group
option can be used to count unique code points, and to compute
the occurrence rate of each one:
When used twice (-gg
) or thrice (-ggg
), the application groups the input by
code point category or code point super category, respectively, which can be used
e.g. for frequency domain analysis:
When --format
is specified exactly as a single char
column: --format=char
,
the application omits all the columns and prints the original file contents,
while highligting each character with a color that indicates its' Unicode
category.
Note that ASCII control codes, as well as Unicode ones, are kept untouched and invisible.
ASCII latin letters (A-Za-z
) are colored in 50% gray color instead of regular
white on purpose — this can be extremely helpful when the task is to find
non-ASCII character(s) in an massive text of plain ASCII ones, or vice versa.
Below is a real example of broken characters which are the result of two operations being applied in the wrong order: UTF-8 decoding and URL %-based unescaping. This error is different from incorrect codepage selection errors, which mess up the whole text or a part of it; all byte sequences are valid UTF-8 encoded code points, but the result differs from the origin and is completely unreadable nevertheless.
While developing the application I encountered strange (as it seemed to be at the beginning) behaviour of Python interpreter, which encoded C1 control bytes as two bytes of UTF-8, while C0 control bytes were displayed as sole bytes, like it would have been encoded in a plain ASCII. Then there was a bit of researching done.
According to ISO/IEC 6429 (ECMA-48),
there are two types of ASCII control codes (to be precise, much more, but for
our purposes it's mostly irrelevant) — C0 and C1. The first one includes ASCII
code points 0x00
-0x1F
and 0x7F
(some authors also include a regular space
character 0x20
in this list), and the characteristic property of this type is
that all C0 code points are encoded in UTF-8 exactly the same as they do in
7-bit US-ASCII (ISO/IEC 646). This
helps to disambiguate exactly what type of encoding is used even for broken byte
sequences, considering the task is to tell if a byte represents sole code point
or is actually a part of multibyte UTF-8 sequence.
However, C1 control codes are represented by 0x80
-0x9F
bytes, which also are
valid bytes for multibyte UTF-8 sequences. In order to distinguish the first
type from the second UTF-8 encodes them as two-byte sequences instead (0x80
→
0xC280
, etc.); also this applies not only to control codes, but to all other
ISO/IEC 8859 code points starting
from 0x80
.
With this in mind, let's see how the application reflects these differences. First command produces several 8-bit ASCII C1 control codes, which are classified as raw binary/non-UTF-8 data, while the second command's output consists of the very same code points but being encoded in UTF-8 (thanks to Python's full transparent Unicode support, we don't even need to bother much about the encodings and such):
The image below illustrates the color scheme developed for the app specifically, to simplify distinguishing code points of one category from others.
Most frequently encountering control codes also have a unique character replacements, which allows to recognize them without reading the label or memorizing code point identifiers: