Paste bug

[ale5000-git]

On notepad I digit a q letter and a tabulation, I copy it and then paste it inside the shell. Instead of pasting the letter and the tabulation it paste quickassist.exe.

Even if I copy printf ' ', on pasting the tabulation character disappear.

[rmyorston]

The console buffer is just a dumb grid of characters and attributes. It doesn't actually contain any tab characters, there are just spaces to fill the gap to the next tab stop.

If you were able to get a tab character into the console buffer it would look like the Unicode character White Circle (U+25CB): ○. Try:

printf '\x09\x0b\x0c\x0e\n'

You'll get something approximating to:

        ♂♀♫

where the tab (\x09) has been displayed as eight spaces. See Code Page 437.

[ale5000-git]

Even if it can't be pasted, I think it isn't good that pasting the tab character (together with others) cause autocompletion to be executed.

[rmyorston]

Sorry, I misunderstood the original issue.

The shell has no way to distinguish between characters typed by the user and characters pasted in. (OK, it could look at how fast the characters appear, but that still wouldn't be conclusive.)

I see the same behaviour with bash on Linux: it happily autocompletes a tab pasted into it.

[ale5000-git]

I'm not sure what is the correct behaviour but apparently the standard command prompt of Windows 10 can distinguish it and ignore all pasted tabs but I don't know how it is done.

[avih]

The solution for this would be bracketed paste, which allows the application (shell here) to distinguish between typed and pasted chars

IIRC bash doesn't enable that by default, and the terminal emulator needs to enable that too. vim also has support, and I think it's also not enabled by default, though no idea if the windows vim supports it too, or any windows application or terminal for this matter.

I'm sure some applications could capture common paste key combos (ctrl+v, etc) and implement a paste which is different than typing, but I don't know how an application can capture paste from the terminal/console's edit-> paste menu item.

[avih]

It does look like the Windows-Terminal (but not the windows console) supports bracketed paste, tested with bash in cygwin, and in bash in linux over ssh.

However, I don't think busybox shell supports it, and also, I'm not sure whether the shell should disable it whenever another application is in the fg (if it doesn't disable it and the other app doesn't support it, then pasting would result in some gibberish surrounding the pasted text).

[ale5000-git]

If you try windows console directly (without opening another shell) in Windows 10 you will see that normal tab cause autocompletion but pasted tab no.

[avih]

TL;DR: I don't think a Windows console application can differentiate typed text vs pasted text.

It has been asked before (no replies) and I googled it too, but so far I couldn't find anything which suggests that a console application can detect paste.

I'm using uclip to copy text to the clipboard and inspect its (text) content:

$ printf "\t" | uclip -i
$ uclip -o | od -c
0000000  \t
0000001
$

• When pasting tab on Windows 7 console it's as if tab was pressed. I.e. it triggers completion, both in cmd.exe and in busybox-w32 interactive shell.
• When pasting tab (possibly among other chars) on Windows 10 console to cmd.exe, then the tab chars are ignored.
• However, pasted tab is not ignored in cmd.exe (and behaves as if tab was pressed, e.g. triggers completion) if the console option "Filter clipboard contents on paste" is unchecked. For me that option was enabled. It's at the leftmost tab "Options" (at the "Edit Options" config block). This option was added to Windows 10 circa 2014 - https://4sysops.com/archives/windows-10-command-prompt-changes/ (and apparently moved out of the "Experimental" tab, and got enabled by default?).
• However No. 2, even when "Filter clipboard contents on paste" is enabled, in busybox interactive sh it's still pasted as if it was typed (even when it was started from the same cmd.exe prompt which just ignored pasted tab).
• In Windows terminal pasted tab appears to be ignored in both cmd.exe and busybox-w32 shell. I don't know whether that's an option I've set in the past, or a default behavior.

From the above, I guessed that maybe pasted content is not filtered at the windows console when the VT mode is enabled at the console (like busybox-w32 interactive shell does by default).

However, setting BB_SKIP_ANSI_EMULATION=0 - which I think is supposed to not engage the console VT mode (either before launching the shell or from within it) does not change it. I.e. pasted tab is still pasted as if tab was pressed (and triggers completion).

Overall, as far as I can tell, it's still a terminal option and not something which a console application can detect programatically (other than hacks which are guaranteed to not always work).

But I still don't know why the option "Filter clipboard contents on paste" seem to not apply in busybox interactive shell, but does apply to cmd.exe.

[rmyorston]

setting BB_SKIP_ANSI_EMULATION=0 - which I think is supposed to not engage the console VT mode

To clarify (perhaps) what busybox-w32 does with VT mode: BB_SKIP_ANSI_EMULATION only affects what happens on output, it doesn't affect how input is handled.

SetConsoleMode takes different flags depending on whether the console handle is for input or output. Skipping internal ANSI emulation sets ENABLE_VIRTUAL_TERMINAL_PROCESSING for output. The corresponding flag for input, ENABLE_VIRTUAL_TERMINAL_INPUT, isn't set by busybox-w32.

[avih]

SetConsoleMode takes different flags depending on whether the console handle is for input or output. Skipping internal ANSI emulation sets ENABLE_VIRTUAL_TERMINAL_PROCESSING for output. The corresponding flag for input, ENABLE_VIRTUAL_TERMINAL_INPUT, isn't set by busybox-w32.

Huh. So I guess that's not the reason that filtering does not seem to work.

Can you reproduce that cmd.exe is correctly affected by the filter option, but busybox-w32 interactive shell always behaves as if filtering is disabled?

[rmyorston]

Can you reproduce that cmd.exe is correctly affected by the filter option, but busybox-w32 interactive shell always behaves as if filtering is disabled?

Yes, that's what I see on Windows 10.

The solution for this would be bracketed paste

FWIW, I've disabled that on Linux. I didn't like the annoying behaviour that ensued. Some Linus Torvalds guy raised a bug report about it.

[avih]

FWIW, I've disabled that on Linux. I didn't like the annoying behaviour that ensued

Yeah, it's not perfect, specifically, AFAIK, a malicious actor can bypass that by including the "disable bracketed paste" escape sequence in the copied text...

However, other than not being able to tame a malicious paste, I do find it useful, and I do like that bash pastes the lines without executing each one (until an actual ENTER is pressed - not pasted), and I do like that vim (once configured) knows to paste pre-formatted text without auto-indenting it, etc.

[rmyorston]

Virtual terminal input is now supported. It's controlled via a new environment variable, BB_TERMINAL_MODE, with the following values:

• 0 Console mode.
• 1 Force terminal mode output.
• 2 Force terminal mode input.
• 3 Force terminal mode input and output.
• 4 Console mode.
• 5 Prefer terminal mode output, fall back to console mode.
• 6 Prefer terminal mode input, fall back to console mode.
• 7 Prefer terminal mode input and output, fall back to console.

The default is 5, which is equivalent to the current BB_SKIP_ANSI_EMULATION=2. Values 1-3 are for testing purposes. The other useful value is 7 which may eventually become the default if this works out.

BB_SKIP_ANSI_EMULATION is still supported with the same values as before. It may be removed in future.

If both variables are set BB_TERMINAL_MODE takes precedence.

[avih]

Virtual terminal input is now supported 1 Force terminal mode output. ...

The title is about input, but all the 1-7 values seem to refer to output. So is this the input "equivalent" of BB_SKIP_ANSI_EMULATION (which affects only output, right?).

I presume that with VT input mode enabled, filtered paste works in bb shell?

EDIT: I now see that some of the 1-7 values refer to output and some to inputand some to both, but I don't think they cover all the combos (9) of prefer-terminal/force-terminal/force-console for both input and output.

[avih]

• 7 Prefer terminal mode input and output, fall back to console.

Two things I noticed in this mode:

• ^C doesn't seem to abort the current interactive input line. Instead, ^C is printed after pressing ENTER. I didn't check whether it has an impact on executing the line.
• VT input mode remains in effect after existing the shell (and going back to cmd.exe). This breaks the UP arrow to recall prior executed commands. Maybe it should be set only when something is reading from the TTY, and unset once it's done? (so e.g. it's unset after pressing ENTER on an interactive shell, but before executing the command).

[rmyorston]

The issue with ^C in shell command line editing has been fixed.

An issue with the shell builtin read not working in an interactive shell has also been fixed.

[rmyorston]

OK, since cmd.exe doesn't understand VT input it's not a good idea to let users enable it "if possible". Supported modes are now:

• 0 Force console mode.
• 1 Force virtual terminal mode for output.
• 2 Force virtual terminal mode for input.
• 3 Force virtual terminal mode for input and output.
• 4 Support virtual terminal input if enabled. Don't alter mode.
• 5 Support virtual terminal input if enabled. Set virtual terminal mode for output, if possible.

The default is 5, as before. It's similar to how Windows works these days. In Windows 11 VT output mode is enabled for both cmd.exe and PowerShell terminals. In Windows 10 and 11 PowerShell supports VT input, if it's enabled.

(BTW, in Windows 10 and 11 cmd.exe resets VT input mode to off when it runs an external command, though not an internal one like dir.)

Value 4 is even more conservative: if VT input mode has been enabled busybox-w32 will handle incoming ESC sequences (like PowerShell) but no changes are made to the terminal mode.

Values 0-3 are only provided for testing purposes. (Though 0 might be useful for rescuing terminals that have got into a weird state.)

[avih]

• 0 Force console mode.

• 1 Force virtual terminal mode for output.

• 2 Force virtual terminal mode for input.

• 3 Force virtual terminal mode for input and output.

• 4 Support virtual terminal input if enabled. Don't alter mode.

• 5 Support virtual terminal input if enabled. Set virtual terminal mode for output, if possible.

How does one disable virtual terminal input, while keeping the output at default (i.e. enabled on windows 10)?

0 forces console on both (?) - which I don't want.

1-3 are different "force VT mode" things - which I don't want.

I don't quite understand what 4 and 5 above actually do.

I did try 4 and it disabled virtual output - which I didn't want.

I also tried 5, and it does keep virtual terminal output, but I don't know in what mode the input is.

Is VT input supposed to be disabled with 5?

[rmyorston]

How does one disable virtual terminal input, while keeping the output at default (i.e. enabled on windows 10)?

If you know the terminal supports VT output (as it does on Windows 10) you can use mode 1.

I did try 4 and it disabled virtual output - which I didn't want.

Are you sure? Mode 4 should just accept the terminal's current VT input and output settings without altering them.

Is VT input supposed to be disabled with 5?

No, in mode 5 busybox-w2 just accepts the current VT input setting of the terminal without altering it. If the terminal supports VT output it will be enabled if it isn't already.

[avih]

How does one disable virtual terminal input, while keeping the output at default (i.e. enabled on windows 10)?

If you know the terminal supports VT output (as it does on Windows 10) you can use mode 1.

Thanks. Does that mean that 1 is actually "Force virtual terminal mode for output and disable VT mode for input"?

And 2 is force VT input and disable VT output?

I did try 4 and it disabled virtual output - which I didn't want.

Are you sure? Mode 4 should just accept the terminal's current VT input and output settings without altering them.

I'm sure that's how I interpreted the result, but apparently the mode is not stateless.

Initially I export BB_TERMINAL_MODE=0 and observed that VT output is disabled (sgrtab --true).

Then I set it to 4 and VT output was still disabled, and that's when I concluded that 4 seem to disable VT output.

So now I started the same and continued with 3 (which enabled VT output) and then 4 (which kept VT output enabled).

So basically, what 4 does seem to depend on the mode it had before?

That's... hard to process...

So which modes depend on the previous mode, and in what ways?

[rmyorston]

Modes 0-3 can be considered a set of flags. The low order bit (bit 0) controls whether VT output is enabled and bit 1 controls VT input.

Mode 4 doesn't alter the terminal settings.

Mode 5 doesn't alter the VT input setting and enables VT output if the terminal supports it.

Whatever the terminal settings busybox-w32 can handle terminal I/O. Previously it was only able to handle output, using either the console API or actual ANSI escape sequences. Now it can do the same for input.

[rmyorston]

Bit 2 of the flag can be considered to mean "enable a VT mode if available".

In an earlier version there were also modes 6 and 7:

Mode 6 would enable VT input if possible while leaving VT output unchanged.

Mode 7 would enable both VT input and output if possible.

[avih]

Modes 0-3 can be considered a set of flags. The low order bit (bit 0) controls whether VT output is enabled and bit 1 controls VT input.

Bit 2 of the flag can be considered to mean "enable a VT mode if available".

These are 3 bits of the whole value, which is 0-7, yes?

Except for mode 0 which explicitly states the word "disable", all other mode ara various forms of "enable".

So what does an unset bit 0/1 mean? if set is enable (VT mode), does unset mean disable VT mode?

Does bit 2 only apply to bits 0/1 when they're set (i.e. bit 2 changes a set 0/1 from force to enable-if-possible)? or only when they're unset, i.e. it changes the unset bits from disabled to "disabled-only-if-unavailable"?

It should be simple, but I just fail to grasp it.

In my mind, input and output VT should each have 3 modes, independently of eachother: force disable, force enable, and enable-if-possible.

Force-disable and enable-if-possible are easy ti grasp (i think).

But what force-enable does is questionable, as it clearly can't force VT on windows 7 for instance.

However, since you already have a mode which presumably forces VT (3), then I presume it's the same as "enable if possible"?

Or maybe, what "force VT" means depends on whether it's input or output.

For output, force VT could mean to disable ansi-emulation even on windows 7, while enable-if-possible would keep ansi emulation on win7, but enable VT and disable ansi emulation on win 10?

And I presume there's something similar with input VT?

But also

Mode 4 should just accept the terminal's current VT input and output settings without altering them. Mode 4 doesn't alter the terminal settings.

So what does it actually do?

Basically, for instance for output, check whether VT mode is enabled at the console, and if yes disable ansi emulation, and if VT mode is not enabled at the console then enable ansi emulation?

And who would set this mode prior to busybox not altering it?

[rmyorston]

Let's see if I can restate this more clearly:

Mode  | Input  | Output
-----------------------
  0   |   C    |    C
  1   |   C    |    V
  2   |   V    |    C
  3   |   V    |    V
  4   |   D    |    D
  5   |   D    |    V

• C turns off VT input or output mode in the terminal. With current Windows console or terminal this will always succeed. It forces the use of the console API.
• V tries to turn on VT input or output mode. This will succeed in Windows terminal and fail in Windows console. In the latter case we fall back to using the console API.
• D detects the input/output mode and uses the appropriate I/O mechanism. It doesn't change the I/O mode.

[avih]

C ... forces the use of the console API.

Thanks. That's what I was missing. So indeed what I wanted was mode 1. Appreciated.

• This will succeed in Windows terminal and fail in Windows console

You mean succeed on Windows 10 (both console and terminal) and fail on earlier windows versions, yes?

• 'D' detects the input/output mode and uses the appropriate I/O mechanism. It doesn't change the I/O mode.

So if busybox doesn't try to set it, who does? You mean for instance some external program which can change the VT I/O modes? (wouldn't the mode be reset once such app exits?)

Also, as for "V", for output, it's easy to observe, for instance on Windows 7 with mode 1 which forces output VT, it will output the escape sequences directly instead of activating the ansi emulation, right?

But for input, what happens e.g. in mode 2 with force input VT mode? Is there some observable broken behavior on windows 7? (I tried, and didn't immediately notice anything broken).

[avih]

By the way, the reason I wanted to disable input VT is that I suspect that vim sometimes gets broken depending on this mode.

Specifically, occationaly (not frequently, not even daily) vim doesn't respond to editing keys (e.g. arrows) right after launching, which forces me to close it. It seems to get fixed after playing with BB_TERMINAL_MODE.

So I wanted to run for a while with input VT disabled to observe if the issue goes away.

What are the advantage of input VT? I.e. why was it enabled by default if possible?

[rmyorston]

You mean succeed on Windows 10 (both console and terminal) and fail on earlier windows versions, yes?

Yes, that did occur to me later. I should have written: This will succeed in Windows terminal and might fail in Windows console.

So if busybox doesn't try to set it, who does?

A previous program? A default selected by the terminal? There must be some I/O setting.

on Windows 7 with mode 1 which forces output VT, it will output the escape sequences directly instead of activating the ansi emulation, right?

Correct. Not all modes are useful. I did suggest 0-3 were mostly for testing purposes.

But for input, what happens e.g. in mode 2 with force input VT mode?

Arrow keys don't work properly, in my experience.

What are the advantage of input VT?

I don't think there's any particular advantage, other than providing support for a valid mode of terminal operation.

Indeed, supporting it requires the upstream code to interpret input sequences to be enabled. Previously we only had console API code for input, now we have both.

Thus providing twice the opportunity for vim to misbehave ;-)

I.e. why was it enabled by default if possible?

It was never enabled by default. And Windows seems not to enable it by default either. At least, not in any circumstances I've encountered.

[avih]

Arrow keys don't work properly, in my experience.

Right, I can reproduce. Previously I didn't try arrows... :D

It was never enabled by default

Right, for some reason I thought the default is 3, but I looked again and it's 5 (try to set output VT, adapt to whatever input VT mode is without changing it).

ndeed, supporting it requires the upstream code to interpret input sequences to be enabled. Previously we only had console API code for input, now we have both.

Thus providing twice the opportunity for vim to misbehave ;-)

This is great!

Thanks again for the explanations and patience :)

[rmyorston]

I've updated the FRP-5007 release notes based on the recent discussion.

Thanks for helping me clarify my thinking!

[avih]

I've updated the FRP-5007 release notes

LGTM.

(I should really sit down one day and try to collect all the windows-specific info - like about BB_TERMINAL_MODE - in one place..., and then open a PR to add it someplace at this repo)

[avih]

Few things regarding BB_TERMINAL_MODE:

The commit message and explanation above support values of 0-5, but the code accepts 6 too.

Windows terminal does support bracketed paste (at least in version 1.20, but probably earlier too). This requires both input and output VT mode (and then enabled with printf \\033[?2004h, and disabled with the same string but ends in l instead of h).

It's currently moot because busybox[-w32] doesn't support bracketed paste, and it breaks the shell input, so it's not a matter of simply enabling it and then expecting to be able to paste literal tab or other chars.

It might be useful to have mode 6 as disable input VT mode and try to enable output VT mode, because some applications might accidentally leave VT input mode, and then some other applications (like cmd.exe) have broken input. So it might be useful to apply the mode whenever control returns to the shell.

If 6 is added as described, not sure whether the default should be 5 or 6.

Relating to the original issue but not to BB_TERMINAL_MODE, bash allows to enter literals control chars after ctrl-v, which can be used to insert a literal tab, e.g.

echo "<ctrl-v><tab>" | xxd<enter>
00000000: 090a 

However upstream busybox doesn't seem to support that, and subsequently neither does busybox-w32, so just abother data point.

[rmyorston]

the code accepts 6

Right, that's an error.

It might be useful to have mode 6 as disable input VT mode and try to enable output VT mode

Isn't that what mode 1 does?

enter literals control chars after ctrl-v

Code to handle ctrl+v was commented out in 2009, though it no longer works. This seems to be a modern replacement (though with minimal testing):

diff --git a/libbb/lineedit.c b/libbb/lineedit.c
index bd4a17348..d73ce86b6 100644
--- a/libbb/lineedit.c
+++ b/libbb/lineedit.c
@@ -3134,15 +3134,16 @@ int FAST_FUNC read_line_input(line_input_t *st, const char *prompt, char *comman
                input_delete(0);
                break;
            }
-//         /* Control-V -- force insert of next char */
-//         if (c == CTRL('V')) {
-//             if (safe_read(STDIN_FILENO, &c, 1) < 1)
-//                 goto return_error_indicator;
-//             if (c == 0) {
-//                 beep();
-//                 break;
-//             }
-//         }
+           /* Control-V -- force insert of next char */
+           if (ic == CTRL('V')) {
+               ic = lineedit_read_key(read_key_buffer, -1);
+               if (ic < 0)
+                   goto return_error_indicator;
+               if (ic == 0) {
+                   beep();
+                   break;
+               }
+           }
            if (ic < ' '
             || (!ENABLE_UNICODE_SUPPORT && ic >= 256)
             || (ENABLE_UNICODE_SUPPORT && ic >= VI_CMDMODE_BIT)

[avih]

It might be useful to have mode 6 as disable input VT mode and try to enable output VT mode

Isn't that what mode 1 does?

I guess it does. I think the different wording about VT out confused me:

1 Force virtual terminal mode for output. 5 .... Set virtual terminal mode for output, if possible.

As if 1 forces VT output mode but 5 only tries to set VT output if possible.

But looking at the table with the C/V/D, 5 is indeed doing the same thing with the output mode as 1 (but 1 disabled VT input while 5 leaves it as is).

So is it applied whenever control returns to the shell?

Knowing that (accidental) input VT can break some apps, do you still think the default should be 5 and not 1? (I'm not sure myself, and I haven't encountered the issue recently).

Code to handle ctrl+v was commented out in 2009, though it no longer works. This seems to be a modern replacement

I can't test right now, but is this working in busybox-w32? if yes, very nice indeed.

[rmyorston]

So is it applied whenever control returns to the shell?

Yes, there's a call to terminal_mode(TRUE) to force a reset at the top of the shell command loop.

Knowing that (accidental) input VT can break some apps, do you still think the default should be 5 and not 1?

I think mode 5 is more likely to let us identify problems with VT input mode (either in busybox-w32 or other apps).

is this working in busybox-w32?

Sort of. The patch above doesn't work upstream. More work is needed.

[avih]

Yes, there's a call to terminal_mode(TRUE) to force a reset at the top of the shell command loop.

Thanks. Nice.

I think mode 5 is more likely to let us identify problems with VT input mode

Can't disagree with that, but as a user, my priorities might be to not get interrupted by issues in other people's code... nevertheless, it is configurable (even if not really accessibly documented), and 5 is not a bad default either way.

Sort of. The patch above doesn't work upstream. More work is needed.

That could be nice, but I wonder to what extent, and whether it would be worth such effort. I'm not sure what the answer is.

For instance it might make it possible to enter literal control code into a string instead of using $'...', or (on unix) to insert the sequence of some editing or function key (where only the initial ESC needs to be inserted literally) instead of figuring out what the sequence is by other means, but on windows, editing and function keys don't generate escape sequence by default (nor anything else which a script can observe, I think), possibly unless input VT is enabled, maybe.

But then, even if we used ctrl-v to insert an up-arrow sequence into a string which we intend to test later in a script which is supposed to handle arrow keys, I don't think busybox-w32 can currently put the terminal into raw mode, so any script which might try to act based on raw editing key sequences input won't work anyway because the input is always buffered and cooked so editing is handled by the console anyway, even in input VT mode, again, I think (I don't have experience with shell scripts which try to handle editing keys on their own, but I think it should be possible on unix in raw terminal mode and dd or some such).

So while cute, I can't determine how much value ctrl-v would have in practice on Windows.

[ale5000-git]

Code to handle ctrl+v was commented out in 2009, though it no longer works. This seems to be a modern replacement (though with minimal testing)

Almost all Windows apps now support CTRL+v and MAIUSC+INS to paste, even cmd.exe of Windows 10. It would be really nice to have it even in BusyBox.

[rmyorston]

Almost all Windows apps now support CTRL+v and MAIUSC+INS to paste

No, the other sort of ctrl+v. From the POSIX sh man page:

<control>-V Insert the next character input, even if the character is otherwise a special insert mode character.

[ale5000-git]

Quote from https://support.microsoft.com/en-us/windows/keyboard-shortcuts-in-windows-dcc61a57-8ff0-cffe-9796-cb9706c75eec:

Ctrl + V (or) Shift + Insert  -  Paste the selected item.
Ctrl + Shift + V  -  Paste as plain text.

If we use Ctrl + V for that, can we use Ctrl + Shift + V for the normal paste? It would be nice to have a shortcut for paste, it is much faster to use keyboard then mouse.

[rmyorston]

See issue #364.