Graphical front-end (GUI-TUI Bridge)

[o-sdn-o]

Terminal users always enter the terminal environment from a graphical environment. The initial interface is always graphical. Until now, we have relied entirely on third-party graphical terminal emulators which significantly limit the capabilities of console applications without providing any way to achieve the required functionality. The only way to overcome these bottlenecks is to create our own bridge from the graphical to the terminal environment.

Our own graphical front-end will open up ways for us to solve the following issues:

• Implementation of full keyboard support. #86
• Displaying fragments of characters (halves), as well as subcell-sized text. https://github.com/directvt/vtm/discussions/72#discussioncomment-949279
• Display of extended grapheme clusters in TUI applications.
• Support for pixel graphics. #164
• Additional character attributes. #165
• Full support for platform-specific clipboard. #273
• Get rid of the 16-color mode in older versions of Windows.
• Support for multiple text cursors and in the correct mode.
• Placing IME in the correct location next to the caret.
• Support for grid lines.

Character Matrix

1x1	2x2	3x1

Each character is a sequence of codepoints (one or more) - this is the so-called grapheme cluster. Using a font, this sequence is translated into a glyph run. The final scaling and rasterization of the glyph run is done into a rectangular matrix of terminal cells, defined either implicitly based on the Unicode properties of the cluster codepoints, or explicitly using a modifier codepoint from the Unicode codepoint range 0xD0000-0xD02A2, the value of which is encoded by the enumeration of “wh_xy” values:

• w: Character matrix width
• h: Character matrix height
• x: Horizontal fragment selector inside the matrix
• y: Vertical fragment selector inside the matrix

Users can explicitly specify the size of the character matrix (by zeroing _xy) or select any fragment (non-zero _xy) of it placing a modifier character from the Unicode codepoint range 0xD0000-0xD02A2 right after the base character (or at the end of the grapheme cluster).

For character matrices larger than 8x4, pixel graphics should be used.

Grapheme Cluster Boundaries

By default, grapheme clustering occurs according to Unicode UAX #29 https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundary_Rules.

To set arbitrary boundaries, the C0 control character ASCII 0x02 STX is used, signaling the beginning of a grapheme cluster. The closing character of a grapheme cluster is always a codepoint from the range 0xD0000-0xDFFFF, which sets the dimension of the character matrix. All codepoints between STX and the closing codepoint that sets the matrix size will be included in the grapheme cluster.

Another brick in the wall

At present only standardized variation sequences with VS1, VS2, VS3, VS15 and VS16 have been defined; VS15 and VS16 are reserved to request that a character should be displayed as text or as an emoji) respectively.

VS4–VS14 (U+FE03–U+FE0D) are not used for any variation sequences

• https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)
• https://www.unicode.org/Public/UNIDATA/StandardizedVariants.txt
• https://www.unicode.org/reports/tr51/tr51-16.html#Direction

So, let's try to play this way:

Glyph run alignment (inside the matrix)

VS	Codepoint	Axis	Alignment
VS4	0xFE03	Horizontal	Left
VS5	0xFE04	Horizontal	Center
VS6	0xFE05	Horizontal	Right
VS7	0xFE06	Vertical	Top
VS8	0xFE07	Vertical	Middle
VS9	0xFE08	Vertical	Bottom

• We are not operating at a low enough level to support justified alignment.
• By default, glyphs are aligned on the baseline at the writing origin.

Glyph run rotation (inside the matrix)

VS	Codepoint	Function
VS10	0xFE09	Rotate 90° CCW
VS11	0xFE0A	Rotate 180° CCW
VS12	0xFE0B	Rotate 270° CCW
VS13	0xFE0C	Horizontal flip
VS14	0xFE0D	Vertical flip

# state - is a three bits integer.
VS10(state) = (state & 0b100) | ((state + 0b001) & 0b011)
VS11(state) = (state & 0b100) | ((state + 0b010) & 0b011)
VS12(state) = (state & 0b100) | ((state + 0b011) & 0b011)
VS13(state) = (state ^ 0b100) | ((state + (state & 1 ? 0 : 0b010)) & 0b011)
VS14(state) = (state ^ 0b100) | ((state + (state & 1 ? 0b010 : 0)) & 0b011)

angle = 90 * (state & 0b011)
hflip = state >> 2

[o-sdn-o]

We need to render a glyph raster of size (MX+2)x(MY+2) with border outer glyphs around the visible raster. When rendering the glyph raster, we need two layers: - Bottom layer for cell contents. - Top layer for outer tails and squiggles of glyphs extending beyond the cell.

Each glyph consists of what falls inside the cell and the tails around it, and they are should be drawn in the following order: - First, the entire first layer with the filled cell bodies is drawn. - Then the layer with the glyph tails is drawn on top of the filled cell bodies.

For the first prototype I will be testing something called Shingled Cell Rendering (a-la SMR for hard drives). For a final image of 1200x1000 pixels for a 120x50 grid with a 10x20 pixel cell, an intermediate buffer of 3600x3000 pixels is required. Each cell stores empty space around itself for hanging elements, increasing its size by three times along each axis, which will be 30x60 pixels per cell.

Shingled Cell Rendering is done in two stages. - First, the background and cores of all cells (10x20) in a row are drawn. - Then, the space around each cell is rendered on top.

The caret, which is a cell-wise SGR attribute, must be rendered as an SGR attribute.

[o-sdn-o]

You can play with a prototype of the vtm graphical window (gui-bridge branch) on various Windows platforms starting with Win8.1 (including Window Server 2019 Core).

If vtm is launched from a graphical shell, or through the start vtm.exe command, then it automatically turns on the graphical mode.

https://github.com/directvt/vtm/assets/11535558/870a5dd7-39aa-4ec0-9a9e-23f03d94ace5

The vtm server starts in the background, despite the fact that its interface is not yet rendered.

[o-sdn-o]

In order to combine the coordinate grid of graphic and text modes, it is necessary, in addition to the integer coordinates of the text cell itself, to add relative fractional coordinates inside the cell in the range [0.0f, 1.0f). In other words, the mouse coordinates when switching from graphic to text mode are converted as follows: fractional_coordinates = cellular_coordinates + (coordinates_inside_the_cell) / cell_size. I think that single precision floating point values ​​should be enough. Now we need to convert all geometric types from int32_t to float32.

This way we will not depend on the cell size and at the same time we will be able to forward the mouse position back to graphic mode if the mouse event was not handled by the text mode.

[o-sdn-o]

We can provide a pixel size of 1x1 2x2 3x3 4x4 etc in addition to the cell size, this is nice for perception and rendering performance.

[o-sdn-o]

According to https://www.unicode.org/ivd/data/2022-09-13/ Variation Selectors in range from 0xE0120 up to 0xE01FF are not used yet. So we can use a sub-range of the Unicode Variation Selectors to specify character dimensions or select its fragment. Four integer values are packed into one byte by plain enumeration "wh_xy".

• w: Matrix width
• h: Matrix height
• x: Horizontal fragment selector inside the matrix
• y: Vertical fragment selector inside the matrix

[o-sdn-o]

Experiments

Github breaks colors on pasted screenshots.
Original:

[o-sdn-o]

2x2 characters:

https://github.com/directvt/vtm/assets/11535558/8a2208eb-a3df-405c-a242-34c229054070

[o-sdn-o]

[o-sdn-o]

Try to output Devanagari (font: "Noto Sans Devanagari"):

https://github.com/directvt/vtm/assets/11535558/d5def070-2241-46b0-881e-c4ec18a0f0f2

[o-sdn-o]

[o-sdn-o]

8x4 character matrix support

https://github.com/directvt/vtm/assets/11535558/60ce28bd-05e6-45e1-896a-a6b43fb05956

[o-sdn-o]

Of course, along with the character matrix, support for both rotation (pi/2, pi, 3pi/2) and flip (hz/vt) is required.

[o-sdn-o]

At present only standardized variation sequences with VS1, VS2, VS3, VS15 and VS16 have been defined; VS15 and VS16 are reserved to request that a character should be displayed as text or as an emoji) respectively.

VS4–VS14 (U+FE03–U+FE0D) are not used for any variation sequences

• https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)
• https://www.unicode.org/Public/UNIDATA/StandardizedVariants.txt
• https://www.unicode.org/reports/tr51/tr51-16.html#Direction

Let's try to play this way:

VS	Codepoint	Function
VS10	0xFE09	Rotate 90° CCW
VS11	0xFE0A	Rotate 180° CCW
VS12	0xFE0B	Rotate 270° CCW
VS13	0xFE0C	Horizontal flip
VS14	0xFE0D	Vertical flip

# state - is a three bits integer.
VS10(state) = (state & 0b100) | ((state + 0b001) & 0b011)
VS11(state) = (state & 0b100) | ((state + 0b010) & 0b011)
VS12(state) = (state & 0b100) | ((state + 0b011) & 0b011)
VS13(state) = (state ^ 0b100) | ((state + (state & 1 ? 0 : 0b010)) & 0b011)
VS14(state) = (state ^ 0b100) | ((state + (state & 1 ? 0b010 : 0)) & 0b011)

angle = 90 * (state & 0b011)
hflip = state >> 2

[o-sdn-o]

Character rotation support

[o-sdn-o]

[o-sdn-o]

[o-sdn-o]

At present only standardized variation sequences with VS1, VS2, VS3, VS15 and VS16 have been defined; VS15 and VS16 are reserved to request that a character should be displayed as text or as an emoji) respectively.

VS4–VS14 (U+FE03–U+FE0D) are not used for any variation sequences

• https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)
• https://www.unicode.org/Public/UNIDATA/StandardizedVariants.txt
• https://www.unicode.org/reports/tr51/tr51-16.html#Direction

Another brick in the wall: Glyph run alignment (inside the matrix)

VS	Codepoint	Axis	Alignment
VS4	0xFE03	Horizontal	Left
VS5	0xFE04	Horizontal	Center
VS6	0xFE05	Horizontal	Right
VS7	0xFE06	Vertical	Top
VS8	0xFE07	Vertical	Middle
VS9	0xFE08	Vertical	Bottom

• We are not operating at a low enough level to support justified alignment.
• By default, glyphs are aligned on the baseline at the writing origin.

[o-sdn-o]

[o-sdn-o]

[o-sdn-o]

https://github.com/directvt/vtm/assets/11535558/5fbc75fe-0c03-4f31-92fd-9740a688d754

[o-sdn-o]

*lines

[o-sdn-o]

[KillyMXI]

Damn. How to prevent this from happening? I have a WT profile for VTM and I only specify path to vtm.exe there as the executable. And after upgrading past 0.9.77 it now opens a separate window instead of WT tab and shows some test screen.

While in prototype stage, this shouldn't be the default behavior like this. It should be opt-in. When done, there should be opt-out cli option regardless of how I call the executable.

[o-sdn-o]

Sorry for this inconvenience. The instant workaround will be to specify cmd /c "path/to/vtm.exe" in the WT profile instead of path/to/vtm.exe.

[o-sdn-o]

In order for vtm to remain within the text console provided to it, it is necessary that this console additionally contains a process such as the command shell cmd..exe or the like.

I don’t yet know any other way to determine the mode required to launch (GUI or TUI), except by looking at the number of processes in the console. If vtm is launched from a graphical environment, the system creates a text console (conhost) for it and places one vtm process there. If vtm is launched from a text environment, then this is usually done from a command shell already running in an existing text console, and therefore the number of processes in this console is more than one.

I'll add an option to override this behavior.

[KillyMXI]

Thanks.

There are times to run console apps from GUI environment and times to run GUI apps from console environment. Can't have a heuristic without overrides.