Allow overriding vim_is_xterm from an option at runtime

[bcat]

Vim uses vim_is_xterm to determine if the terminal is one for which the builtin xterm termcap should be applied. Right now this includes xterm, rxvt, screen (when $TERM is set to something like screen.xterm-256color), and a couple other terminals. However, tmux's $TERM strings (e.g., tmux-256color) are not currently included in this list, although tmux also emulates an xterm-like terminal.

Without Vim detecting tmux as an xterm-like terminal, things mostly work, but any non-standard Vim termcap extensions are not set, even though tmux actually supports (most of?) them. For example, t_RV isn't set, and so v:termresponse doesn't get populated. Likewise, bracketed paste doesn't work without manually setting the relevant four termcap codes in your own Vim configuration.

You can work around this by setting tmux's $TERM to screen-256color, but this causes other problems since tmux's own terminfo has diverged from screen long ago.

It should be trivial to include tmux in the vim_is_xterm function, but I wanted to check that this was sane first. Unlike screen, tmux doesn't seem to have the ability to pass through the underyling terminal's $TERM (e.g., there is no tmux.xterm-256color terminfo like there's a screen.xterm-256color one).

However, I'm not sure this is really a problem in practice. tmux's terminfo entries are basically screen's, but with more xterm features. IMO, treating tmux like an xterm will be positive for almost all users, and the few folks who use tmux with an underyling terminal that isn't compatible with xterm are probably going to have a bad time regardless of what Vim does.

[bcat]

Issue #1830 seems related, but is probably overkill. Vim doesn't include a custom builtin termcap for rxvt (instead just treating it as xterm) even though rxvt has way more differences from xterm than screen has from xterm.

If we're gonna do something fancy, it'd be neat if the builtin termcaps and accompanying logic were configurable rather than hardcoded in Vim at compile time, but that feels much larger in scope than adding one line to os_unix.c. :)

[bcat]

I was thinking of this a bit more in the context of #6687, and I think what I really want is an option (settable from Vimscript) for the list of $TERM strings Vim treats as xterm-like for the purposes of initializing the termcap and related state. Hardcoding this list into Vim source makes it hard to change, and means that you only get useful defaults for a bunch of Vim termcap entries if your $TERM is "xterm" or one of a select few variants. This causes problems for terminal emulators that are mostly xterm-like, but that should really have their own terminfo entry for capabilities that don't match real xterm.

It'd be nice to make other hardcoded lists like use_xterm_like_mouse configurable too, IMO, but I'm not sure that's as important.

[bcat]

For example, to work around the linked issue, I'd configure mintty to properly set $TERM=mintty and then I'd add this to my ~/.vimrc

if &term =~# '\vmintty%(-|$)'
  " Proposed new option:
  set xtermlike

  let &t_AU = "\e[58:5:%dm"
  let &t_8u = "\e[58:2::%lu:%lu:%lum"
endif

This would override only the two escape sequences are problematic for mintty, while still using the rest of Vim's builtin xterm termcap for bracketed paste, and while still using the normal Vim logic for parsing termresponse, etc.

One potential gotcha is that in order to be useful, setting (or clearing) xtermlike would have to overwrite termcap variables with ones from Vim's builtin termcaps, request and reprocess termresponse, etc. I think that's reasonable, but it may be surprising for setting an option to have this behavior. (For instance, in my code snippet above, if I incorrectly set xtermlike after modifying t_AU and t_8u, my changes to those termcap variables would simply be overwritten.) But it seems that setting the term option in a running Vim instance already reloads the termcap in this way, so this new option would just share that behavior.

[brammool]

Doesn't this work:

if &term =~# '\vmintty%(-|$)' set term=xterm-color let &t_AU = "\e[58:5:%dm" let &t_8u = "\e[58:2::%lu:%lu:%lum" endif

[bcat]

It works for my particular problem today, but I believe the set term=xterm-color will overwrite any differences between mintty's own (system) terminfo and xterm's. It also means that non-Vim apps in :terminal windows will see $TERM=xterm-color and not $TERM=mintty. So it seems to run at cross purposes to the desire that different terminals should have different terminfo and not claim to be xterms.

In general, I'm all for setting $TERM to a "truthful" setting and filing bugs/sending patches to get the system terminfo database up to date. But I'm trying to figure out how to make that work well with Vim's builtin termcaps, which set extensions for various things that are not standardized in upstream terminfo yet. Realistically, Vim's defaults work well for most things that are xterm-like, with only an override or two needed, so it'd be lovely if I could say to Vim "use your builtin termcap for your extensions, but respect the system termcap for my actual terminal otherwise".

I'm also aware this all sucks and ideally everything should come from terminfo, full stop. :) But since Vim is already in the business of patching up and extending the system terminfo, I'm just wondering if that can be a little more flexible (to make it easier to eventually upstream things like this into the real terminfo database).

[bcat]

(Basically, I have these lines in my config to set the things Vim's builtin termcap provides for xterm. I can keep updating these from time to time. It's not that big a deal. It'd just be cleaner IMO if there was a single option I could use instead.)

[brammool]

Jonathan Rascher wrote:

It works for my particular problem today, but I believe the set term=xterm-color will overwrite any differences between mintty's own (system) terminfo and xterm's. It also means that non-Vim apps in :terminal windows will see $TERM=xterm-color and not $TERM=mintty. So it seems to run at cross purposes to the desire that different terminals should have different terminfo and not claim to be xterms.

If you do have a terminfo for mintty, then what do you need the Vim xterm defaults for? You could do: let save_term = &term set term=xterm-color let entry1 = &t_XX let entry2 = &t_YY let &term = save_term let &t_XX = entry1 let &t_YY = entry2

Not very nice, but can work.

In general, I'm all for setting $TERM to a "truthful" setting and filing bugs/sending patches to get the system terminfo database up to date. But I'm trying to figure out how to make that work well with Vim's builtin termcaps, which set extensions for various things that are not standardized in upstream terminfo yet. Realistically, Vim's defaults work well for most things that are xterm-like, with only an override or two needed, so it'd be lovely if I could say to Vim "use your builtin termcap for your extensions, but respect the system termcap for my actual terminal otherwise".

Perhaps what you want is to additionally use the xterm defaults for termcap/terminfo settings that are not set from terminfo/termcap?

I'm also aware this all sucks and ideally everything should come from terminfo, full stop. :) But since Vim is already in the business of patching up and extending the system terminfo, I'm just wondering if that can be a little more flexible (to make it easier to eventually upstream things like this into the real terminfo database).

The problem is that the termcap/terminfo system is too limited, it doesn't take care of any runtime options that terminals provide. You would have to create a new entry for every combination of build options, command line flags and config files.

I tried to work around that by introducing the mechanism to ask the terminal for the entries. Only xterm supports that. Problem with that is that it takes some time to get the responses, and if the user in the mean time starts a shell then the codes go the wrong way.

Best would be some kind of "side channel" to contact the terminal and query the escape codes it gives for keys and the escape codes it accepts for output. That might work for a local terminal, but many use tmux/screen and ssh, and especially then it's useful, but then we don't have the side channel and the current mechanism can be a bit slow.

Perhaps some kind of caching could be used, to fetch the codes first time a terminal (with specific options) is used, and then next time just pass the checksum. It's a bit like a complicated $TERM name.

Vim             Terminal

{request checksum}      ->
                <-  {termcap checksum}
Lookup in cache
hit: use cached termcap
miss: {request termcap}  ->
             <- {termcap entries}

Implementing that would require support from all popular terminals.

-- If Microsoft would build a car... ... Occasionally your car would die on the freeway for no reason. You would have to pull over to the side of the road, close all of the car windows, shut it off, restart it, and reopen the windows before you could continue. For some reason you would simply accept this.

/// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\ /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\ \\ an exciting new programming language -- http://www.Zimbu.org /// \\ help me help AIDS victims -- http://ICCF-Holland.org ///

[bcat]

In general, I'm all for setting $TERM to a "truthful" setting and filing bugs/sending patches to get the system terminfo database up to date. But I'm trying to figure out how to make that work well with Vim's builtin termcaps, which set extensions for various things that are not standardized in upstream terminfo yet. Realistically, Vim's defaults work well for most things that are xterm-like, with only an override or two needed, so it'd be lovely if I could say to Vim "use your builtin termcap for your extensions, but respect the system termcap for my actual terminal otherwise".

Perhaps what you want is to additionally use the xterm defaults for termcap/terminfo settings that are not set from terminfo/termcap?

This is precisely my use case, yes. I use a few different xterm-like terminals (primarily hterm on Chrome OS, mintty on Windows, and urxvt on Linux). Vim provides default versions of a bunch of its own termcap extensions for xterm (and urxvt), and these defaults work almost entirely consistently across the all the terminals I use. The problem is I'd like to use the terminals' own $TERM values and terminfo entries when possible (because they are not all perfect clones of xterm), but then I lose all of the Vim-specific termcap extensions.

I can manually specify them in my ~/.vimrc. It's only a few dozen lines. I just thought it might be nice if Vim let me get those builtins automatically, by making the hardcoded list of xterm-like terminals in os_unix.c more configurable.

[brammool]

Jonathan Rascher wrote:

In general, I'm all for setting $TERM to a "truthful" setting and filing bugs/sending patches to get the system terminfo database up to date. But I'm trying to figure out how to make that work well with Vim's builtin termcaps, which set extensions for various things that are not standardized in upstream terminfo yet. Realistically, Vim's defaults work well for most things that are xterm-like, with only an override or two needed, so it'd be lovely if I could say to Vim "use your builtin termcap for your extensions, but respect the system termcap for my actual terminal otherwise".

Perhaps what you want is to additionally use the xterm defaults for termcap/terminfo settings that are not set from terminfo/termcap?

This is precisely my use case, yes. I use a few different xterm-like terminals (primarily hterm on Chrome OS, mintty on Windows, and urxvt on Linux). Vim provides default versions of a bunch of its own termcap extensions for xterm (and urxvt), and these defaults work almost entirely consistently across the all the terminals I use. The problem is I'd like to use the terminals' own terminfo entries when possible (because they are not all perfect clones of xterm), but then I lose all of the Vim-specific termcap extensions.

I can manually specify them in my ~/.vimrc. It's only a few dozen lines. I just thought it might be nice if Vim let me get those builtins automatically, by making the hardcoded list of xterm-like terminals in os_unix.c more configurable.

Just taking all entries that aren't set yet is likely to cause problems once Vim adds an entry that the specific terminal doesn't support.

What perhaps would work is to add a function that adds specific entries.

call getfromtermcap("xterm", ['8f', 'AB'])

A bit more flexible but also more verbose:

let &t_8f = gettermcap("xterm", '8f')

-- hundred-and-one symptoms of being an internet addict:

185. You order fast food over the Internet

     /// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\ /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\ \\ an exciting new programming language -- http://www.Zimbu.org /// \\ help me help AIDS victims -- http://ICCF-Holland.org ///

[bcat]

Sorry for my lack of response on this! I recently updated tmux, and I noticed it offers an interesting solution: a "terminal features" list that enables you to selectively enable particular features without needing to deal with particular escape sequences. So, for example, when TERM=hterm-256color (in terminfo, but lacking extensions in both Vim and tmux), instead of manually entering a lot of escape sequences, you can just enable a short list of "features".

It's a little more high level than the gettermcap suggestion, but still avoids the problems of blindly copying all the XTerm termcap extensions.

What do you think of something like this, conceptually?

[brammool]

Hmm, tweaking features makes it even more difficult for Vim to figure out the right escape sequences. This problem already exists for xterm, which has both compile time options and a runtime configuration. For xterm I worked with Thomas Dickey on a request-reply format to have the terminal tell Vim what its escape sequences are. Since terminfo entries are static, this should work better. This also works over an ssh connection. The big disadvantage is the startup delay. And when Vim executes a shell command, the reply may end up with that command instead of with Vim. It's not perfect. I have been wondering to improve on this by just having one big reply that Vim can wait on before running a shell command. I/O should be fast enough these days to handle a large reply quickly. It's like reading the terminfo entry from the terminal itself.