Add support for VT102 Insertion-Replace Mode (IRM)

[markmi]

Insertion-Replacement Mode (IRM)

ESC  [   4   h
033 133 064 150

Set selects insert mode and turns INSERT on. New display characters move old display characters to the right. Characters moved past the right margin are lost.

ESC  [   4   l
033 133 064 154

Reset selects replace mode and turns INSERT off. New display characters replace old display characters at cursor position. The old character is erased.

---

original issue body follows.

The following is from recording command line editing in an ssh session to a freebsd environment (via FreeBSD's script). The displayed text is odd in Windows Terminal. First I show a textual form that shows the escape sequences and such in a readable form (I eliminated bell-ring characters from the file and other lines of text):

more ~/odd_commnd_line_editing_display.txt ls ~/sys_typescriptsESC[1ESC[4h-ESC[4lESC[4hlESC[4lESC[4hTESC[4lESC[4hdESC[4lESC[4htESC[4lESC[4h ESC[4

A normal display in other environments (for example macOS terminal doing an ssh into FreeBSD) looks like:

cat ~/odd_commnd_line_editing_display.txt ls -lTdt ~/sys_typescripts/

(The above shows the actual ls command that was executed.)

But under Windows Terminal (Preview) it shows:

cat ~/odd_commnd_line_editing_display.txt ls -lTdt ~ypescripts/

This is also how it ends up looking during the command line editing, not showing the actual text of the command to be executed: missing some text.

The sequence is from deciding to add "-lTdt" after having typed the path. (It was a "set -o vi" type of command line editing context.)

I was using TERM being xterm-256color on FreeBSD .

[j4james]

I suspect this is because the Windows terminal doesn't yet support Insertion-Replacement Mode (IRM).

[DHowett-MSFT]

Interesting! Didn't know IRM was a thing. I'm going to repurpose this as a task to implement IRM.

Notes:

Insertion-Replacement Mode (IRM)

ESC  [   4   h
033 133 064 150

Set selects insert mode and turns INSERT on. New display characters move old display characters to the right. Characters moved past the right margin are lost.

ESC  [   4   l
033 133 064 154

Reset selects replace mode and turns INSERT off. New display characters replace old display characters at cursor position. The old character is erased.

[egmontkob]

Yet another symptom of the same bug:

Resize the window to 63 columns. Start mc (Midnight Commander) on Ubuntu, compiled against the S-Lang screen drawing library (as it's done in Ubuntu).

Expected: The bottom row should display 9 buttons, each of them being 2 characters for the function key number + 5 for the action. The rightmost should be <space>9Pu~Dn, reaching all the way to the very end.

Actual: <space>9Pu~D is printed with the expected colors, and the last column remains empty, showing the terminal's background color.

What the S-Lang library does is an ancient and obsolete hack for some terminals that don't have "pending wrap" (@j4james do you know more about them?), to print in the lower right corner without the cursor automatically wrapping to the next line and shifting everything up. It prints the intended last letter (n in this case) in the penultimate (62nd) column, and then using IRM, prints the penultimate letter (D) again in the penultimate (62nd) column, expecting it to push the previously printed n to the 63rd.

Even at bigger window sizes, such as the normal 80 columns, mc misses to paint the bottom right corner using some of its skins, e.g. F9 -> Options -> Appearance -> sand256 is one of these. For some mysterious reason, S-Lang decides to use different drawing method depending on some property of the skin. In this case, it repeats the aforementioned trick with space characters, whereas with most other themes, it just emits the escape sequence that wipes to the end.

This is a tiny inconsistency (almost-bug) somewhere in S-Lang, my wild guess is that it could be related to non-color attributes (in this case italic) being used there, but I don't think it's worth investigating S-Lang's behavior any further. Also, for the record: I couldn't reproduce the issue with mc compiled against ncurses, apparently it knows about "pending wrap" and knows that this hack is not necessary.

[j4james]

What the S-Lang library does is an ancient and obsolete hack for some terminals that don't have "pending wrap" (@j4james do you know more about them?)

I know DOS and the original Windows shell didn't support "pending wrap", but then they wouldn't have supported IRM mode either. However, it wouldn't surprise me if there were other third-party terminal emulators for Windows that shared that wrapping behaviour, but were still able to handle IRM, so maybe that's the sort of thing that S-Lang is targetting.

By the way, IRM is on my TODO list (assuming nobody else takes it on first). It's just that we don't yet have a handler for ANSI modes, and the existing dispatch mechanism makes that a bit of a pain to add, so that was something I was hoping to address first. There are various other issues related to parameter handling (e.g. #2101 and possibly #942), and I wanted to make sure we had something that could satisfy all of those requirements if possible, so it's probably not going to be a quick fix.

[zaphod77]

this is a a real issue. pico editor seems to use this when ran under emulated xterm. and the characters do not move to the right like they are supposed to.

still an issue as of this day.

[henrikj242]

I'm also affected by this. Is there any known way to work around it?

[j4james]

I was actually looking into this recently, and I think it's going to require some architectural decisions before we can consider implementing anything.

I'd initially thought that the AdaptDispatch class could handle the inserting prior to writing anything, by simply scrolling the line it was about to write into by the appropriate amount. But that assumes it knows how much space the text is going to take (which it doesn't), and if the text is going to wrap onto another line, then potentially handle inserting on later lines (which may not even exist yet if vertical scrolling is required).

As things currently stand, I don't think this is possible. So either the inserting must be handled in the PrintString implementation itself (which is essentially the WriteCharsLegacy function that we don't want to touch). Or the AdaptDispatch class needs to take over the majority of the text layout, and have PrintString be a very basic operation that only writes to a single line (no inserting, wrapping, scrolling, or margin checks).

Considering the additional complications that are going to be introduced to the layout as we expand the VT functionality, I think the latter option may well be the best long term approach. But that is going to require some additional APIs, e.g. to query how much space a string of text will occupy and/or where it would be clipped.

[philkernick]

I also have the same problem, but I have a workaround.

FreeBSD uses termcap (rather than terminfo) and the termcap entries for xterm include the 'im' and 'ei' capabilities which invoke the VT102 IRM. In the Linux terminfo database, these capabilities are not implemented.

The workaround is to create a .termcap file in your home directory with this one entry in it: xterm-256color:mi@:IC=\E[%d@:ei@:ic=\E[@:im@:tc=xterm-new:

This will use this instead for the xterm-256color terminal (which is what Windows Terminal is), and delete the 'im' and 'ei' capabilities.

Until the IRM capabilities are implemented, this works perfectly.

[DHowett]

It's wild how IRM works with line wrapping in xterm (351). Text is pushed off the right margin, as expected, but when the inserted region is wider than will fit it wraps and pushes text on the next line to the right.

This is a terribly specific feature. I wonder why it was introduced!

[vixie]

no version of xterm has ever not supported IRM. if you're not going to implement it, please identify yourself as a VT100 instead of xterm256-color. the pending-wrap thing is a side show, don't worry about it. just please make :ei=\E[4l: and :im=\E[4h: work. i've gone back to konsole as a result of this bug.

[zaphod77]

I have to use putty because of this STILL, even though putty doesn't have perfect xterm emulation either. For those of you who can't seem to reproduce this, your pico is actually nano, which has a workaround for this bug.

If you actually use the pico text editor, you can very easily see the behavior. it's the simplest test case.

[mreymann]

The workaround is to create a .termcap file in your home directory with this one entry in it: xterm-256color:mi@:IC=\E[%d@:ei@:ic=\E[@:im@:tc=xterm-new: Until the IRM capabilities are implemented, this works perfectly.

Thanks, your workaround fixes some weird behaviour when connecting to FreeBSD boxes. Waiting for a real fix ;-)

[vixie]

Marc Reymann writes:

...

Thanks, your workaround fixes some weird behaviour when connecting to
FreeBSD boxes. Waiting for a real fix ;-)

freebsd is not the only place this matters. other systems also expect that
if the offered terminal type is xterm256-color, then IRM will work. my
workaround is to use TERM=vt100, which means, my workaround is to use
konsole instead.

[zatarain]

Great!! I was dealing with this bug since April, it was really annoying. I just apply the workaround with xterm-clear:te=\E[?1049l:ti=\E[?1049h:mi@:IC=\E[%d@:ei@:ic=\E[@:im@:tc=xterm-new: and it worked!! I hope Microsoft implements the real fix soon!!

[DHowett]

I'm taking a crack at this.

It looks like we don't even have SM/RM, so there's a lot of plumbing to do.

[DHowett]

@j4james the approach I've gone with -- which is of course subject to discussion and eventual change :smile: -- is to make TextBuffer do an inserting write at the cursor position. I've not made any architectural changes to Adapt or change whose job it is to do text layout. I tested gnome-terminal here & found that they don't do any re-wrapping, even if the line was emitted wrapped. Insert mode there impacts only the physical display & not the logical paragraph/wrap state.

[vixie]

happy to review or test.

[j4james]

make TextBuffer do an inserting write at the cursor position.

Yeah, but where are you doing that? In the WriteCharsLegacy implementation? And if so, that assumedly requires another OutputMode flag, or at least another WC_* flag. I just assumed we wouldn't want to be introducing more complexity there?

I was starting to think maybe now was a good time to take on #780, and incorporate the IRM functionality there. That seemed easier than worrying about how this would impact the legacy code. But maybe I'm overthinking things and you've got a simpler solution that I'm just not seeing.

It looks like we don't even have SM/RM, so there's a lot of plumbing to do.

Yeah, that's why I wanted to get #7799 out the way first. It should at least be easer to extend the framework now than it was back then.

[zatarain]

It would be nice if you add some spaces to see how it behaves on multi word lines.

I'm taking a crack at this.

It looks like we don't even have SM/RM, so there's a lot of plumbing to do.

[DHowett]

@j4james oh, right. Sorry to be inexact. Since all of the at-cursor writes coming out of WriteCharsLegacy (and any other source in conhost; see _stream.cpp:536 for the only instance I could find) go through SCREEN_INFORMATION::Write(OutputCellIterator), I put IRM determination in SCREEN_INFORMATION and kept it out of WCL. That keeps it away from any modes, control character handling, knowledge of backup/line wrapping state, etc.

Every other Write uses a coordinate target, but perhaps it would be prudent to rename this one to WriteAtCursor.

Does that seem a reasonable place?

[j4james]

OK that's brilliant. That's so much simpler than I was expecting. I can't see any reason why that wouldn't work. :shipit:

[DHowett]

Happy New Year! Draft PR up in #8689.

[zaphod77]

Please ignore results in Gnome Terminal, unless they can be reproduced under real XTerm.

IRM is designed for the case of a text editor with autowrap turned off. This saves bandwidth. The test case animated graphic is not a real world scenario, though it is what sending the raw characters to the terminal would do.

[kingma-sbw]

Connecting to a (Arch)Linux box has the problem. No IRM. If I isseu export TERM=vt100 vi behaves as designed. It would be great to include this as setting. Connecting with PuTTY works fine and offers a variety of settings:

[Forge36]

Is this the root cause of cursor mode not changing? IE: in CMD vs

Or should this be considered a new issue?

[DHowett]

The feature here--IRM--is unrelated to any CMD features or console applications that use the "Windows-style" cooked input feature (Python, PowerShell without PSReadline, Command Prompt).

[kingma-sbw]

I'm also affected by this. Is there any known way to work around it?

Yes it is setting the TERM variable to VT100

export TERM=vt100

[davhae]

I'm also affected by this. Is there any known way to work around it?

I am just using tmux to prevent insert mode before the last space in a command. The Issue only happens to me when editing a command from the history.

export TERM=vt100 did not change this behaviour.

[Anutrix]

Can't SSH into a lot of lab/course(from online course sites) VMs because of this.

Any workarounds for this? export TERM=vt100 didn't work.

[vixie]

Numan Zaheer Ahmed wrote on 2022-12-11 15:32:

Can't SSH into a lot of lab/course(from Coursera, Udemy, etc.) VMs because of this.

Any workarounds for this?

don't use microsoft's terminal emulator. stick with securecrt, or install linux or freebsd inside "hyper-v" and run your terminal sessions inside the client VM. neither microsoft's "terminal" or "ssh" are ready for prime time.

|export TERM=vt100| did't work.

i should have been clearer. "export TERM=vt100" is what you do on the remote system after you log in. the sshd that launches your shell there is going to pass the TERM variable it gets from the ssh session. i have not figured out how to get this to be passes as "vt100" by the invoking ssh inside the powershell or command CLI on the initiating side. but if i override it on the remote shell, everything i invoke there hears that my terminal type is "vt100".

i guess coursera and udemy don't have shells and the TERM variable their sshd gets from your ssh has to be correct? sorry about my ignorance, i don't use ssh that way.

-- P Vixie

[Anutrix]

don't use microsoft's terminal emulator. stick with securecrt, or install linux or freebsd inside "hyper-v" and run your terminal sessions inside the client VM. neither microsoft's "terminal" or "ssh" are ready for prime time.

It's ssh on WSL2(which is basically a VM now) so not Microsoft ssh. As for windows terminal, not using it is not solution. Good alternative console emulators/terminals are hard to find on Windows.

|export TERM=vt100| did't work. i should have been clearer. "export TERM=vt100" is what you do on the remote system after you log in. the sshd that launches your shell there is going to pass the TERM variable it gets from the ssh session. i have not figured out how to get this to be passes as "vt100" by the invoking ssh inside the powershell or command CLI on the initiating side. but if i override it on the remote shell, everything i invoke there hears that my terminal type is "vt100". i guess coursera and udemy don't have shells and the TERM variable their sshd gets from your ssh has to be correct? sorry about my ignorance, i don't use ssh that way. … -- P Vixie

I ran the command inside the remote system after I logged in. Still didn't work. Thx anyways. I should have mentioned where I ran it.

[vixie]

Numan Zaheer Ahmed wrote on 2022-12-11 15:58:

...

It's ssh on WSL2(which is basically a VM now) so not Microsoft ssh. As for windows terminal, not using it is not solution. Good alternative console emulators/terminals are hard to find on Windows.

if you're using WSL2 you might try mintty which claims to run there and is based on putty which has working xterm escape sequences like IRM.

I ran the command inside the remote system after I logged in. Still didn't work. ... Thx anyways. I should have mentioned where I ran it.

can you say more about what doesn't work and under what conditions? i find that once i set TERM=vt100 (or TERM=vt220) on the remote shell, all the text editors and screen oriented utilities stop trying to use IRM, and thus work passably well with Windows Terminal.

i'm using Windows Terminal Preview but i just tested and it's the same for Windows Terminal as shipped from the microsoft store.

-- P Vixie

[Anutrix]

if you're using WSL2 you might try mintty which claims to run there and is based on putty which has working xterm escape sequences like IRM.

Recent release of mintty binary seems hard to find(and am trying to avoid MSYS2). I will try wsltty from the developer of mintty instead. Update: Same issue in mintty. Starting to think it might be WSL/Windows issue. Additionally, mintty is used in Git Bash terminal where WSL doesn't seem to work anymore. Gives no output when I try to start WSL.

I ran the command inside the remote system after I logged in. Still didn't work. ... Thx anyways. I should have mentioned where I ran it. can you say more about what doesn't work and under what conditions?

I just SSH into the remote shell. I press and hold a character(let's say 'a' or just paste a long string). It overwrites the first character after end of line is reached instead of moving to next line.

[vixie]

depending on the remote shell choice, it may not notice that you've changed the TERM variable. more reliably, the commands you run from that shell will inherit the modified environment (TERM=vt220 or whatever). to be sure you're getting shell-level behaviour effects warranted by your new terminal type, do something like this:

export TERM=vt220 exec bash

or, if you're like me using csh:

setenv TERM vt220 exec tcsh

or if you're using a modern mac/os ssh server-side:

export TERM=vt220 exec zsh

and so on. but your best test is still:

export TERM=vt220 emacs -nw $somefile

if your TERM indicates IRM and your terminal doesn't implement IRM then emacs (among other utilities) will not be able to keep your screen updated.

Numan Zaheer Ahmed wrote on 2022-12-12 13:04:

[Anutrix]

export TERM=vt220 exec bash

It works fine inside new shell but I lose UI benefits as it goes to a basic terminal.

and so on. but your best test is still: export TERM=vt220 emacs -nw $somefile if your TERM indicates IRM and your terminal doesn't implement IRM then emacs (among other utilities) will not be able to keep your screen updated.

The remote shell is a limited environment without vim, nano, emacs. There might be a problem with the shell itself. I'll try check with their support.

Update: Sry about all that. There PS1 seems to be broken/bad on the remote shell. So ignore my earlier troubles. Wasn't related to VT102 IRM. Marking my messages as off-topic.

[dmiller423]

This is still open 3&&1/2yrs later? @lhecker I see you've been assigned this 2weeks ago, is it actually something in progress finally?

[zaphod77]

The main issue here is UnicodeStorage, which is not very friendly to trying to do this. #8000 needs to finish first, and then this can be solved.

[lhecker]

I've recently removed UnicodeStorage, however it doesn't actually make my implementation all that much more trivial. This is because my more important and more long term goal is to fully support Unicode (and extended grapheme clusters in turn) for conhost and Windows Terminal. Implementing IRM requires an "text insertion" primitive which may be implemented based on our existing code base, however this would build up on our existing non-Unicode support, which I'm really hesitant to do. I'd basically be writing code that I'm actively trying to remove. This is why I'm trading a bit longer development time for IRM, to write those new Unicode-supporting primitives that we will need in other areas as well, with a better integration into our long term vision.

That said, I've already begun working on this. :) I'm hoping to finish it before the 1.17 release ships, but it's possible I might miss it due to (unrelated) personal circumstances. In either case however I'm confident to finish it within January.

[ghost]

:tada:This issue was addressed in #14700, which has now been successfully released as Windows Terminal Preview v1.17.1023.:tada:

Handy links:

• Release Notes
• Store Download

[vixie]

works! tyvm.