consider allowing stylized input ala html/markdown

[dankamongmen]

This opens a huge can of worms, but it's worth thinking about idly: in what way could Notcurses accept marked up input, i.e. Markdown or (very limited) HTML? Are we the right place to do it? How would it tie in to existing output functionality, if at all?

ncplane_puthtml(n, "hey <b>asshole</b> this is pretty sweet");

is a lot more compact than

ncplane_putstr(n, "hey ");
ncplane_style_on(n, NCSTYLE_BOLD);
ncplane_putstr(n, "fat boy, asshole");
ncplane_style_off(n, NCSTYLE_BOLD);
ncplane_putstr(n, "i want to kill you");

and allows you to do things like alignment of such text, whereas that would be a massive pain in the ass currently. also note that there is at least one bug in the second version (missing space between asshole and i).

definitely post-3.0 material.

[tomek-szczesny]

If ncplane_puthtml simply calls other ncplane procedures, I can't see how it could be dangerous.

As per our discussion #1831, parsing terminal control codes could be accomplished using this. I'd argue that accepting ANSI control actually should come first, because it's by far the easiest to do, usually translating escape sequences directly into a set of notcurses procedures.

With HTML, I'd argue it's not that simple. Keeping track of the hierarchy of the tags is one thing. Trying to emulate the expected behavior using limited capabilities of a terminal, that's another. Actually, so few HTML tags may be directly translated into terminal realm that perhaps bbcode instead makes more sense. At least the coverage would be almost complete, and not "very limited". ;)

And Markdown, totally! No questions asked.

[dankamongmen]

oh it would be a very, very restricted subset of html, probably less powerful than markdown. i don't want to introduce layout logic etc. whatsoever, but allow those HTML capabilities already trivially exposed by Notcurses to be written with HTML markup.

which is why i want to know what all VTspeak you intend to handle. basically as soon as cursor movements are allowed, we need a cooperating or at least controlled (in the sense of Ophiocordyceps) generator of material. i envisioned the HTML capability to primarily be used with string literals, not, like, a web browser. just handling SGR and colors is no problem whatsoever.

[dankamongmen]

which is why i want to know what all VTspeak you intend to handle. basically as soon as cursor movements are allowed, we need a cooperating or at least controlled (in the sense of Ophiocordyceps) generator of material. i envisioned the HTML capability to primarily be used with string literals, not, like, a web browser. just handling SGR and colors is no problem whatsoever.

we don't technically need one, just if we don't want the output to look like ass

[dankamongmen]

if you wanted to prepare code to convert basic VTspeak to ncplane_() actions, i support it, and look forward to the PRs. i just also want to think about how we control the inferred geometry of whatever's providing this input. but that's wholly orthogonal to the initial work of basic SGR conversions. and that latter's a good way for you to build up some trust. Nie mój cyrk, nie moje małpy; i think you can take it from here!

[dankamongmen]

oh, and since i haven't made it obvious enough: i am thrilled and thankful that you're interested in moving on this! Dziękuję!

[tomek-szczesny]

I wouldn't worry that much about the geometry. If the user inputs crap, they'll get crap in return. I think that as long as user knows exactly how big ncplane is, they should be able to provide compliant data that fits the box.

So let's say I'll try to implement ncplane_putvt(n, ssh_channel.read());

Nie ma sprawy, as long as you're willing to spend half an hour on providing feedback to my code rather than tossing it away in the end. ;)

[tomek-szczesny]

One thing worth noting, though: As far as Usage goes, notcurses currently has no functions that lets a user select colors from the 4-bit FG/BG palette. Sure that's hardly ever necessary for new software, BUT... from the VT perspective, each terminal may and will have its own definitions of colors from this palette. Personally I use terminal.sexy to develop my own palette, and most ncurses soft appears to depend on it as well, heck, even toilet --gay does.

It's probably a desirable feature for those who want their notcurses-based programs be fully supported by Linux tty. But as far as my understanding of inner affairs go, supporting that won't be entirely straightforward.

For now I think we'll get away with converting 4-bit colors into 24-bit counterparts from VGA palette to provide basic VT support, but a new issue shall address that problem the future. Note that HTML also defines a few fixed colors that ought to be converted into 4-bit codes.

[dankamongmen]

One thing worth noting, though: As far as Usage goes, notcurses currently has no functions that lets a user select colors from the 4-bit FG/BG palette. Sure that's hardly ever necessary for new software, BUT... from the VT perspective, each terminal may and will have its own definitions of colors from this palette. Personally I use terminal.sexy to develop my own palette, and most ncurses soft appears to depend on it as well, heck, even toilet --gay does.

we support an 8-bit palette through the ncpalette functionality, which one could certainly restrict to the lower 16 colors. or am i missing something? this also allows palette reprogramming.

[tomek-szczesny]

Oh yeah, it is indeed implemented to some extent. What is missing is plane API function to switch fg or bg color into one from 4-bit or 8-bit palette. I can only see this:

int ncplane_set_fg_rgb8(struct ncplane* n, int r, int g, int b);
int ncplane_set_bg_rgb8(struct ncplane* n, int r, int g, int b);

My understanding is that either of the aforementioned functions would fall back to 8-bit or 4-bit palette if that what's available. My intention is to force color selection directly from either of these palettes, if the original VTspell explicitly said so.

[tomek-szczesny]

In other words, what shall we do when we parse the following SGR:

/x1b/x5b/x20/x6d

It just says "green foreground, please".

[dankamongmen]

In other words, what shall we do when we parse the following SGR: /x1b/x5b/x20/x6d It just says "green foreground, please".

so there are two questions here, a larger one and a smaller one:

• that doesn't necessarily mean SGR. it does on an ANSI terminal, sure, and most terminals are ANSI, sure, but the larger question remains: what interface do we want to present to the user? if we're strictly doing ANSI, that ought IMHO be reflected in the API. there are two cases: presentation of already-generated text, and an active subprocess generating text. just as a string is meaningless without specifying its encoding, a string of control codes is meaningless without specifying the control sequence interface against which it was generated (in practice, what terminfo database entry was selected).

assuming ANSI for now, there are three different things we could do:

• blindly call ncplane_set_fg_palindex(n, 2), hoping it is green
• program the palette, and translate 2 to the green we want
• translate 2 to an RGB color and use ncplane_set_fg_rgb()

because does the user truly mean "green", or do they mean "palette entry 2"?

[tomek-szczesny]

Well... To be honest my focus is on ANSI, and xterm-supported extensions, at least until we hit any discrepancy in the process. And surely enough terminfo shall reflect the capabilities as we extend them.

My wild guess is "green" and "palette entry 2" are the same thing, although I admit I don't know which standard governs that (Not 1979 ANSI, I guess?).

But anyway, yes, my question was what to do with request for foreground color # 2 and I think ncplane_set_fg_palindex(n, 2) is the most sensible answer.

I wasn't aware this function exists, it's not explained in USAGE.

Okay, so what about 256-color mode escapes? ESC[38;5;#m I think translating them into 24-bit RGB color is the way to go.

[dankamongmen]

Well... To be honest my focus is on ANSI, and xterm-supported extensions, at least until we hit any discrepancy in the process. And surely enough terminfo shall reflect the capabilities as we extend them.

understood re: ANSI, just trying to get everything we'll one day need do out in front, so we can design the API properly.

My wild guess is "green" and "palette entry 2" are the same thing, although I admit I don't know which standard governs that (Not 1979 ANSI, I guess?).

the user might have changed their palette. and even among the ansi colors there's drastic divergence in exact rgb: https://en.wikipedia.org/wiki/ANSI_escape_code#3-bit_and_4-bit

i mean, just saying ncplane_set_fg_palindex(2) is likely to get a green. and if not, presumably the user knows what they're doing (maybe they have seizures if they see green, idk). so that's probably the right thing to do. just exploring the space.

Okay, so what about 256-color mode escapes?

likewise ncplane_set_fg_palindex() if the number of supported colors equals or exceeds 256 (tcache.caps.colors).

if it doesn't, things become more interesting. i'd need to go look at the ncpalette logic. it's designed for you to be able to use 8-bit palette indices, and i quantize down or juggle palette reprogramming or something. i don't think that's actually been written, though. i do know i quantize RGB down to 256 or 8 colors. so, your options would seem to be:

• define a palette, map the upper 238 colors to rgb values, and translate to ncplane_set_fg_rgb8() calls, relying on raster-time quantization, or
• use ncpalette and when it doesn't work, yell at me, and i'll do it at some point (this would only affect environments with fewer than 256 colors, so while it's important, it's not, like, sacrifice-a-child important), or
• quantize yourself (please don't; i don't want quantization spread all over the place), or
• tell them to fuck themselves with their superANSI 256 colors, what kind of first-world shit is this

i feel that the second (use of ncpalette) is the correct thing to do. my goal in notcurses is always to allow full functionality, and weaken the output for boned-out terminals. just as i allow rgb on an 8- or 16- or 256-color terminal, we ought allow 256-entry palettes to be provided, and then knocked down at runtime.

maybe there's actually even code there! i don't think so, though.

[tomek-szczesny]

Okay now I'm seriously confused, but that might just be early morning coffee deficiency.

My understanding is that depending on which SGR is used, the color is selected, either from 3-bit palette, 4-bit palette, 8-bit palette or full 24-bit spectrum. There is no ambiguity between these escape sequences, we always know exactly which palette entry user wanted, and from which palette. Each palette is a subset of a bigger palette. Color from the 4-bit subset generally may be customized within the terminal, but I'd argue that colors outside this subset are meant to follow the definition (hence the structure of 8-bit palette).

I think what we need is an access to 8-bit palette through ncplane_set_*_palindex(); One function to rule them all. The reasoning is the following: 3/4-bit palettes are subsets sitting under first 16 entries of 8-bit, easy enough to convert from original numbering, and may be handled "natively" on notcurses side to preserve terminal settings. All the following entries of 8-bit palette shall be converted to RGB and handled by quantization thingie. A few sentences of explanation in USAGE et voila, all palettes are supported.

Regarding your concern about full flexibility offered to the user: Yes, the user will be able to use any color they want, be it through RGB, or 4-bit codes to preserve the terminal's internal understanding of it.

The risk involved is that some terminals may actually have their own custom 8-bit palettes that are not defined as Wikipedia says. I think this is so unlikely I'd leave this bit for future fix if it ever happens. There is no risk of changes in API though.

And I'm officially raising my voice at you, as the VT parser logic is getting pretty mature - already past state machine brainstorming, stack overflow and segfault stage, I might actually get there throughout this weekend. I'll need this issue dealt with shortly, and in the meantime I'll use mockups provided by @chromafunk.

[tomek-szczesny]

Just to keep everyone hyped, here's the output of toilet --gay Fi I got so far. For any reason, I get two abundant newlines for now, but I'll get to the bottom of this tomorrow.

[dankamongmen]

My understanding is that depending on which SGR is used, the color is selected, either from 3-bit palette, 4-bit palette, 8-bit palette or full 24-bit spectrum. There is no ambiguity between these escape sequences, we always know exactly which palette entry user wanted, and from which palette.

as far as i'm aware, an SGR setaf or setab works one of two ways: palette-indexed or 8bpc RGB. palette-indexed is an index into a palette size defined by the terminal; a number greater than the palette size is either a no-op or undefined behavior or taken modulo the size or who knows. but there's no difference between a 3-bit palette, a 4-bit palette, or an 8-bit palette control sequence AFAIK.

Color from the 4-bit subset generally may be customized within the terminal, but I'd argue that colors outside this subset are meant to follow the definition (hence the structure of 8-bit palette).

this certainly seems reasonable, but nonetheless the palette can be reprogrammed all the way.

I think what we need is an access to 8-bit palette through ncplane_set_*_palindex();

this is exactly what you ought do. it should work fine.

[tomek-szczesny]

I think what we need is an access to 8-bit palette through ncplane_set_*_palindex();

this is exactly what you ought do. it should work fine.

Well, any chance of implementing this function then? ;)

[dankamongmen]

Well, any chance of implementing this function then? ;)

i must have missed something. what is needed beyond the existing ncplane_set_*_palindex()? i'm in the matrix room if that would be more effective.

[tomek-szczesny]

Woohoo, Utf-8 and 4bit color codes are already supported! Thanks to @chromafunk for helping me out with tests.

I learned malloc today! :D

Anyway, I'll dig into the palindex implementation, because honestly I still don't know what this function does and how to use it to emit 8bit palette.

[dankamongmen]

hey @tomek-szczesny , just checking in. you still on a lengthy eurostyle vacation? is that where @joseluis is too? y'all suddenly disappeared on me. hope all's well =].

[tomek-szczesny]

Hi! Not really, recently got shitloads of actual work that pays my bills. I'm not giving up on vt thingie just yet, but it is going to take some time at the current rate.

[dankamongmen]

Hi! Not really, recently got shitloads of actual work that pays my bills. I'm not giving up on vt thingie just yet, but it is going to take some time at the current rate.

totally understood!

[joseluis]

lengthy eurostyle vacation? is that where @joseluis is too? y'all suddenly disappeared on me. hope all's well =].

hello there Nick. I'm well, you know I like to switch every few months what I work on. I'm currently enjoying the August heatwave while studying and working on different projects. I noticed you were involved a lot on Windows & Mac related stuff lately, so it felt natural to me not being very involved. I'll come back when the wind changes again, like with mary poppings :p

I have to say I certainly admire your consistency and ability to stay focused on notcurses every single day. I don't know how you do it but massive respects for you man.

[dankamongmen]

hello there Nick. I'm well, you know I like to switch every few months what I work on. I'm currently enjoying the August heatwave while studying and working on different projects. I noticed you were involved a lot on Windows & Mac related stuff lately, so it felt natural to me not being very involved. I'll come back when the wind changes again, like with mary poppings :p

all good! thanks for getting the rust compiling again =] yeah that makes total sense, just wanted to make sure i hadn't pissed anyone off or anything like that =]

[tomek-szczesny]

Nah, it takes getting used to your workflow but we're cool. The last time I met an asshole in the open source community, that encounter prompted me to rewrite his shit software from scratch. I guess you can relate to some extent.