Graphical front-end (GUI-TUI Bridge)

[o-sdn-o]

• [ ] Keyboard-only access, #86
• [x] Character Matrix Unicode Character Geometry Modifiers
• [x] Extended grapheme clusters
• [ ] Pixel graphics, #164
• [ ] Background color diffusion, #165
• [ ] GUI mode
  • [x] Win32
  • [ ] X11
  • [ ] macOS
  • ~Wayland~
• [x] Multiple text cursors
• [x] TSF integration
• [x] Multi-focus
• [ ] Hyperlinks
• [ ] Sound

[o-sdn-o]

We need to render a glyph raster of size (MX+2)x(MY+2) with border outer glyphs around the visible raster. When rendering the glyph raster, we need two layers: - Bottom layer for cell contents. - Top layer for outer tails and squiggles of glyphs extending beyond the cell.

Each glyph consists of what falls inside the cell and the tails around it, and they are should be drawn in the following order: - First, the entire first layer with the filled cell bodies is drawn. - Then the layer with the glyph tails is drawn on top of the filled cell bodies.

For the first prototype I will be testing something called Shingled Cell Rendering (a-la SMR for hard drives). For a final image of 1200x1000 pixels for a 120x50 grid with a 10x20 pixel cell, an intermediate buffer of 3600x3000 pixels is required. Each cell stores empty space around itself for hanging elements, increasing its size by three times along each axis, which will be 30x60 pixels per cell.

Shingled Cell Rendering is done in two stages. - First, the background and cores of all cells (10x20) in a row are drawn. - Then, the space around each cell is rendered on top.

The caret, which is a cell-wise SGR attribute, must be rendered as an SGR attribute.

[o-sdn-o]

You can play with a prototype of the vtm graphical window (gui-bridge branch) on various Windows platforms starting with Win8.1 (including Window Server 2019 Core).

If vtm is launched from a graphical shell, or through the start vtm.exe command, then it automatically turns on the graphical mode.

https://github.com/directvt/vtm/assets/11535558/870a5dd7-39aa-4ec0-9a9e-23f03d94ace5

The vtm server starts in the background, despite the fact that its interface is not yet rendered.

[o-sdn-o]

In order to combine the coordinate grid of graphic and text modes, it is necessary, in addition to the integer coordinates of the text cell itself, to add relative fractional coordinates inside the cell in the range [0.0f, 1.0f). In other words, the mouse coordinates when switching from graphic to text mode are converted as follows: fractional_coordinates = cellular_coordinates + (coordinates_inside_the_cell) / cell_size. I think that single precision floating point values ​​should be enough. Now we need to convert all geometric types from int32_t to float32.

This way we will not depend on the cell size and at the same time we will be able to forward the mouse position back to graphic mode if the mouse event was not handled by the text mode.

[o-sdn-o]

We can provide a pixel size of 1x1 2x2 3x3 4x4 etc in addition to the cell size, this is nice for perception and rendering performance.

[o-sdn-o]

According to https://www.unicode.org/ivd/data/2022-09-13/ Variation Selectors in range from 0xE0120 up to 0xE01FF are not used yet. So we can use a sub-range of the Unicode Variation Selectors to specify character dimensions or select its fragment. Four integer values are packed into one byte by plain enumeration "wh_xy".

• w: Matrix width
• h: Matrix height
• x: Horizontal fragment selector inside the matrix
• y: Vertical fragment selector inside the matrix

[o-sdn-o]

Experiments

Github breaks colors on pasted screenshots.
Original:

[o-sdn-o]

2x2 characters:

https://github.com/directvt/vtm/assets/11535558/8a2208eb-a3df-405c-a242-34c229054070

[o-sdn-o]

[o-sdn-o]

Try to output Devanagari (font: "Noto Sans Devanagari"):

https://github.com/directvt/vtm/assets/11535558/d5def070-2241-46b0-881e-c4ec18a0f0f2

[o-sdn-o]

[o-sdn-o]

8x4 character matrix support

https://github.com/directvt/vtm/assets/11535558/60ce28bd-05e6-45e1-896a-a6b43fb05956

[o-sdn-o]

Of course, along with the character matrix, support for both rotation (pi/2, pi, 3pi/2) and flip (hz/vt) is required.

[o-sdn-o]

At present only standardized variation sequences with VS1, VS2, VS3, VS15 and VS16 have been defined; VS15 and VS16 are reserved to request that a character should be displayed as text or as an emoji) respectively.

VS4–VS14 (U+FE03–U+FE0D) are not used for any variation sequences

• https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)
• https://www.unicode.org/Public/UNIDATA/StandardizedVariants.txt
• https://www.unicode.org/reports/tr51/tr51-16.html#Direction

Let's try to play this way:

VS	Codepoint	Function
VS10	0xFE09	Rotate 90° CCW
VS11	0xFE0A	Rotate 180° CCW
VS12	0xFE0B	Rotate 270° CCW
VS13	0xFE0C	Horizontal flip
VS14	0xFE0D	Vertical flip

# state - is a three bits integer.
VS10(state) = (state & 0b100) | ((state + 0b001) & 0b011)
VS11(state) = (state & 0b100) | ((state + 0b010) & 0b011)
VS12(state) = (state & 0b100) | ((state + 0b011) & 0b011)
VS13(state) = (state ^ 0b100) | ((state + (state & 1 ? 0 : 0b010)) & 0b011)
VS14(state) = (state ^ 0b100) | ((state + (state & 1 ? 0b010 : 0)) & 0b011)

angle = 90 * (state & 0b011)
hflip = state >> 2

[o-sdn-o]

Character rotation support

[o-sdn-o]

[o-sdn-o]

[o-sdn-o]

At present only standardized variation sequences with VS1, VS2, VS3, VS15 and VS16 have been defined; VS15 and VS16 are reserved to request that a character should be displayed as text or as an emoji) respectively.

VS4–VS14 (U+FE03–U+FE0D) are not used for any variation sequences

• https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)
• https://www.unicode.org/Public/UNIDATA/StandardizedVariants.txt
• https://www.unicode.org/reports/tr51/tr51-16.html#Direction

Another brick in the wall: Glyph run alignment (inside the matrix)

VS	Codepoint	Axis	Alignment
VS4	0xFE03	Horizontal	Left
VS5	0xFE04	Horizontal	Center
VS6	0xFE05	Horizontal	Right
VS7	0xFE06	Vertical	Top
VS8	0xFE07	Vertical	Middle
VS9	0xFE08	Vertical	Bottom

• We are not operating at a low enough level to support justified alignment.
• By default, glyphs are aligned on the baseline at the writing origin.

[o-sdn-o]

[o-sdn-o]

[o-sdn-o]

https://github.com/directvt/vtm/assets/11535558/5fbc75fe-0c03-4f31-92fd-9740a688d754

[o-sdn-o]

*lines

[o-sdn-o]

[KillyMXI]

Damn. How to prevent this from happening? I have a WT profile for VTM and I only specify path to vtm.exe there as the executable. And after upgrading past 0.9.77 it now opens a separate window instead of WT tab and shows some test screen.

While in prototype stage, this shouldn't be the default behavior like this. It should be opt-in. When done, there should be opt-out cli option regardless of how I call the executable.

[o-sdn-o]

Sorry for this inconvenience. The instant workaround will be to specify cmd /c "path/to/vtm.exe" in the WT profile instead of path/to/vtm.exe.

[o-sdn-o]

In order for vtm to remain within the text console provided to it, it is necessary that this console additionally contains a process such as the command shell cmd..exe or the like.

I don’t yet know any other way to determine the mode required to launch (GUI or TUI), except by looking at the number of processes in the console. If vtm is launched from a graphical environment, the system creates a text console (conhost) for it and places one vtm process there. If vtm is launched from a text environment, then this is usually done from a command shell already running in an existing text console, and therefore the number of processes in this console is more than one.

I'll add an option to override this behavior.

[KillyMXI]

Thanks.

There are times to run console apps from GUI environment and times to run GUI apps from console environment. Can't have a heuristic without overrides.

[o-sdn-o]

Georgian script sample

[alphaqueueop]

is there possible setting or bind to reset the zoom in gpu accident zoom with mousewheel too much?

[o-sdn-o]

There is no such option at the moment. I will make shortcuts and mouse wheel configurable for zoom functions and such in #86.

Now I can add, for example, CapsLock+0 shortcut to reset cell scaling.