explore a general scheme for acquiring terminfo data from the terminal directly

[dankamongmen]

XTerm supports XTGETTCAP to request terminfo data. Experiment with it and other terminals to see whether or not critical terminfo escapes can be generally acquired. If so, we could eliminate our dependence on a correct TERM. Indeed, we could eliminate our dependency on the terminfo library entirely, and be guaranteed up-to-date information.

[WSLUser]

Would the DECRQSS settings query also possibly work instead? Better to support DEC over XT sequences when possible.

[dankamongmen]

Would the DECRQSS settings query also possibly work instead? Better to support DEC over XT sequences when possible.

unfortunately no. i need terminfo strings (e.g. "what do i write to make things bold"?). decrqss is a different set of capabilities: https://vt100.net/docs/vt510-rm/DECRQSS.html

[WSLUser]

Was just a thought. I assumed settings query to mean what you can do with the terminal, which it does but in a different way than I realized.

[kovidgoyal]

Note that kitty supports XTGETTCAP and I am a big fan of using it if you are anyway doing query roundtripping. Indeed kitty extends XTGETTCAP to allow you to query things not covered by terminfo (which is completely ossified anyway). https://sw.kovidgoyal.net/kitty/kittens/query_terminal/

[dankamongmen]

yep, kitty would work very well with this scheme.

[dankamongmen]

so a few things to think about:

• not every terminal will support this. to retrieve everything we want, we'd need send close to one hundred XTGETTCAP requests, at about 10B each, for almost 1KB of requests. our current request string is about 100B, so this would all still fit in a single 1500B ethernet frame. the replies are larger, however, and might well push us into two-frame territory. perhaps we could limit our requests to a subset of the terminfo keys we look up? but which ones?
• so we'd still need to consult the terminfo database for terminals which don't implement this, which we'd likely want to do in parallel, before we know whether the terminal in question is going to respond. we'd need some kind of resolution mechanism. worst case would be someone who set TERM incorrectly, such that there were many disagreements. we'd obviously prefer what the terminal gives us, assuming it gives us anything...
• this doesn't involve the terminfo intermediary, so this could be a good way to introduce capabilities not yet (ever) covered by terminfo. that might indeed be the most attractive element about it.

right now setupterm() is called before we start the input engine (and begin parsing potential replies). i'd like that to be in parallel, but we need to know the escape sequences covered by terminfo before starting the input engine, so i'm not sure whether that's ever possible. it doesn't seem so =/.

[kovidgoyal]

Do you actually need a 100 different things? I suggest you do a bit of differential analysis on the common terminals you support to see what they actually differ on (which is very few things these days). You will likely get a set of dozen or so. If TERM is any of those terminals use a smaller query, on other less common/esoteric/legacy terminals, use a larger query.

In other words I am suggesting you craft the query based on an internal database of TERM values.

[dankamongmen]

Do you actually need a 100 different things? I suggest you do a bit of differential analysis on the common terminals you support to see what they actually differ on (which is very few things these days). You will likely get a set of dozen or so. If TERM is any of those terminals use a smaller query, on other less common/esoteric/legacy terminals, use a larger query.

yep, that's about what i'm planning.

In other words I am suggesting you craft the query based on an internal database of TERM values.

hrmmm so a dynamic query. interesting. the thing i don't like about this is that one of the major points is to eliminate problems, where possible, caused by an incorrect TERM.

i think further analysis requires a quantitative differential analysis of the existing terminfo database.

[kovidgoyal]

On Mon, Oct 18, 2021 at 10:27:34AM -0700, nick black wrote:

Do you actually need a 100 different things? I suggest you do a bit of differential analysis on the common terminals you support to see what they actually differ on (which is very few things these days). You will likely get a set of dozen or so. If TERM is any of those terminals use a smaller query, on other less common/esoteric/legacy terminals, use a larger query.

yep, that's about what i'm planning.

In other words I am suggesting you craft the query based on an internal database of TERM values.

hrmmm so a dynamic query. interesting. the thing i don't like about this is that one of the major points is to eliminate problems, where possible, caused by an incorrect TERM.

Well, you can still largely do that, since one of the queries you can make is for the term name, and if the user has set a different TERM you can then do a second round if needed. This will penalize users who set incorrect TERM values a little, but hopefully that hack will become rarer with time.

i think further analysis requires a quantitative differential analysis of the existing terminfo database.

Indeed.

[dankamongmen]

also hpa, which is of critical importance, and which seems to be missing from the Kitty FreeBSD termcap

[dankamongmen]

this could resolve #2541 for good and immediately, and hpa is certainly important to us.

[dankamongmen]

tcap_cb() could really stand some cleanup (and unit testing).