Backslash, then Ctrl-B produces unexpected results in Emacs mode

[nickpapadonis]

Test case A:

1. Type backslash
2. Type Ctrl-B
3. Carrot B appears
4. The expected result is to go backwards one character, instead of printing Carrot B

There is been discussion that this behavior is expected based on p106 of Korn's book. At the same time typing backslash and Ctrl-C works as expected without printing Carrot C. Backslash Ctrl-D behaves the same as Test case A.

There is a video reproducing this at: https://www.youtube.com/watch?v=ojfAgdnwQzw&feature=youtu.be

Notes from KSH mailing list discussion: Kurtis: That behavior is present in the ksh93u+ release and thus is not something introduced since the most recent stable release seven years ago. Note that the unexpected behavior is due to the backslash. Without the char under the cursor being a backslash [ctrl-B] and [ctrl-F] behave as you would expect. This affects pretty much every control char such as [ctrl-A] to move to the start of the line. Whether this behavior is good or bad is debatable. If you feel it should be changed, despite being long standing behavior, please open an issue

The problem isn't the presence of parenthesis; it's the backslash. Type a\[ctrl-B]. This behavior only happens if the most recent character you typed is a backslash. So you can type a()b, then move the cursor inside the parens, type \[ctrl-B] and the same thing will happen. Personally, I think the magic backslash is a misfeature. You can always type [ctrl-V][ctrl-B] to insert a literal [ctrl-B]. Feel free to open an issue. Even better, create a patch for us to review.

On that platform /bin/ksh is 93t+ 2010-03-05 and exhibits the behavior you described. You should be able to run /bin/ksh --version to figure out which ksh variant you're using.

Having said all that I agree with you the behavior is confusing and should be changed regardless of how many years it has been in effect.

[saper]

@nickpapadonis There are some bits missing in the way you describe how to reproduce the issue, probably something got cut out by the markdown editor

[nickpapadonis]

Fixed the comment thanks

[jghub]

I am using vi mode so never noticed this behaviour. I now have looked it up in the bolsky/korn book (@nickpapadonis: if you like and care for ksh I can really recommend that you get a hardcopy (long out of print, used samples you can get for 5$ or so from abebooks/amazon)) and it proofs that the behaviour you see is explicitly intended. from page 106:

\ Escapes next character. Editiing characters and the user's Erase, Kill, and interrupt characters may be entered in a command line or search string if preceded by \. The \ removes the next character's editing features, if any.

this seems clear enough to me. I disagree with krader's "Having said all that I agree with you the behavior is confusing and should be changed regardless of how many years it has been in effect."

while it is correct, that control chars like ^B might also be inserted via ^V^B this convenience feature for emacs mode users has to be kept in place and the issue then could be closed as "not-a-bug" I believe. @dannyweldon @marcastel (and everybodye;)): what do you think?

[marcastel]

This is also documented with the same wording in the man page (src/cmd/ksh93/sh.1):

\ Escape next character. Editing characters, the user's erase, kill and interrupt (normally ^?) characters may be entered in a command line or in a search string if preceded by a . The \ removes the next character's editing features (if any).

I concur with @jghub in that this appears compliant. I am not an emacs user either.

[saper]

It also works this way in the vi mode as well without any editing mode at all.

There is one problem though - the documentation says this is a "command", so I'd expect that typing `` (backslash) and then some character quotes it. So far so fine.

However, it seems to work like a permanent marker though: just type backslash, a, b to get \ab. Then press backspace once (\a), again (\) and again (^? in my case - my erase character).

Is the "command" something that is there permanently in the line buffer or is this just a keystroke pressed?

[jghub]

oh, you are right regarding also happens in vi mode! it never happened to me since I have added emacs-like cursor movement keybindings to my vi mode so that I can use ^B and ^F from within vi insert mode. and in this combo \^B actually just backspaces :).

regarding your question: it seems to come about because you never leave input mode (in vi, I mean, in emacs no difference...) so you delete the b than the a. at this point your whole input is \ and the next keystroke looses its special meaning -- including del (ascii octal 177). and when you hit del the \del combination inserts the \0177 char. seems consistent, no?

[jghub]

so I have learned something new. I never was aware of \ doing more than protecting symbols special to the shell. I completely missed the line editor support to input control chars in this way...

[nickpapadonis]

I update the issue as some characters were lost. There are many comments above. I'm confused, is this behavior a feature? What does the feature enable the user to do?

[jghub]

yes, it is a "feature" since explained in the book and the manpage. it can also be confusing, obviously, since the two builtin editors (vi and emacs) in ksh thus treat \ specially, not as an ordinary character.

what it allows, is to easily enter verbatim those control-chars into string definitions. if you type \^B it is interpreted as "insert ascii character with octal code 002 (stx)" (i.e. the control-B character) instead of "move cursor one character to the left". it just is not what you expected to happen ...

[nickpapadonis]

What page is this described in the book? I have it. Thanks

[jghub]

p. 106

[saper]

at this point your whole input is \ and the next keystroke looses its special meaning

I have mistakenly used the word "command" which means something different in the Korn shell. This is an "edit command". "All edit commands operate from any place on the line (not just at the beginning)". So in that way '\' is just a keystroke (like ^A moving the cursor) but it leaves some visual indication the next character is about to quoted.

An '\' already entered in the buffer is a different thing in my opinion. It just sits there.

Regarding ^V - ksh uses this as "Display version of the shell", it works in the emacs/gmacs mode as well as in the command mode of vi mode. However, it is also a default value of the lnext capability of the terminal discipline - this means the next character (keystroke!) is about to be entered literally.

This can be changed or turned off using stty lnext ..., at least on FreeBSD, this leaves `\' edit command as the only way to enter them.

[jghub]

An '' already entered in the buffer is a different thing in my opinion. It just sits there.

I see your point but I am not averse to the present behaviour (which also might be justified). but I only now have noted that the two editors behave different in this situation. vi does what you describe (and what is ok behaviour with me) while emacs does what you seem to prefer in this situation (just treat \ as ordinary character in the buffer when deleting)

maybe this difference is also documented and thus a 'feature' (did not check), but probably not. in which case this is simply inconsistent behaviour and it would be desirable to change it in either vi or emacs mode.

[nickpapadonis]

stty lnext ...

Did not resolve the issue in Debian 10

Did not resolve the issue in FreeBSD 12.1

Did I enter the correct command?

I am still unclear on how this feature is useful. What does it provide? I.e. typing \ (back slash) then Ctrl-B shows ^B. Is the intension to type \ (back slash), then some control characters?

[saper]

stty lnext is not a solution. I was referring to this:

You can always type [ctrl-V][ctrl-B] to insert a literal [ctrl-B].

[ctrl-v] is not a feature of ksh, it is a feature of a so-called "line discipline" that controls the terminal behaviour in UNIX. It can be turned off or assigned to a different key with stty lnext.

To answer your question:

The way I understand this is the intention - to add control characters that would otherwise do something else, similar to Ctrl-V with default setting of most terminals.

[nickpapadonis]

I noticed that the behavior works with other characters other then Ctrl-B. For instance backslash then Ctrl-D.

Some questions: Should Ctrl-B be removed from the possible control characters that can be entered after a backslash.? Potentially only in interactive mode. Thereby giving precedence to the Emacs editing of the buffer? Unsure if the book describes precedence of input characters in their interactive editing mode.

Can we conclude that typing backslash then a control character is a feature based on p106 of the book David Korn authored?

Thanks

[jghub]

I noticed that the behavior works with other characters other then Ctrl-B. For instance backslash then Ctrl-D.

yes. as already stated in my first reply: "\ Escapes next character". any character that is. those with no special meaning. those special to the shell. those special to the editor. other control characters.

Some questions: Should Ctrl-B be removed from the possible control characters that can be entered after a backslash.? Potentially only in interactive mode. Thereby giving precedence to the Emacs editing of the buffer? Unsure if the book describes precedence of input characters in their interactive editing mode.

I would say "no" since one cannot predict whether overall users prefer the emacs "meaning" of ^B (users of vi mode obviously don't...) or want consistent means to input all control characters.

and, more important, \ escapes all characters. it would seem totally wrong to exclude one or two of those (^B,^F) from this treatment "just" because they happen to have a meaning to the chosen editor mode.

I understand that the behaviour (can't backspace with ^B over \ immediately after typing it) irritated you but it seems tolerable to now remember it and use the left arrow key for a change if the situation happens again (or switch to vi mode ;)).

Can we conclude that typing backslash then a control character is a feature based on p106 of the book David Korn authored?

I believe the manpage together with the book is the de facto "specification" of the KornLanguage. in this special case, as @marcastel already pointed out in this thread, it is explained not only in the book but also in the manpage in identical words. so it is not a random thing. it is fully intentional and defined behaviour.

in view of all this, my proposal would be to close this issue, right?

Thanks

[nickpapadonis]

If one types backslash and Ctrl-C, Ctrl-C is not output to the screen with a carrot in front of it. In this case Ctrl-C is captured for it's special meaning. If one types backslash and Ctrl-D, Ctrl-D is output to the screen and it is not captured for it's special meaning. In this sense, I would argue that the behavior is non-uniform for at least Ctrl-C. Does this make sense?

[nickpapadonis]

Can someone add the lines from Korn's book p106 that justify this behavior? I have my book in storage. Thanks

[posguy99]

I'm just a ksh user, but here's the page.

Learning the Korn Shell, 2nd ed dances around the same issue beginning on page 27.

[nickpapadonis]

Wow there it is. I agree we can close this issue out as expected behavior. Thanks for all the input!

[saper]

maybe this difference is also documented and thus a 'feature' (did not check), but probably not. in which case this is simply inconsistent behaviour and it would be desirable to change it in either vi or emacs mode.

From what I can see this is not documented and might be a bug indeed.

Looks like a no-editor mode (neither vi not emacs) does behave the same as vi. I prefer emacs behavior. But we should not break the functionality on totally dumb terminals.

[saper]

Also found backspacing mode in vi which makes me stuck forever behind the backslash:

• Start editing in vi mode
• After backslash type something then ESC to switch to command mode
• Use shift-T, backslash command to get behind the backslash
• Use shift-C to replace the text till the end of line
• Type something in the input mode
• Backspace all out - you’ll be stuck “forever” behind the backslash

Note: in the vi’s change command it is not allowed to backspace to the left where one has started

Weird, but consistent with how the change command behaves...

[dwierenga]

oh, you are right regarding also happens in vi mode! it never happened to me since I have added emacs-like cursor movement keybindings to my vi mode so that I can use ^B and ^F from within vi insert mode. and in this combo \^B actually just backspaces :).

Sorry to comment on a closed issue, but is there any chance you could share those keybindings? Maybe start a wiki page in this project? There's a general lack of information around the web for the esoteric way to get KSH to interpret key strokes.

[marcastel]

@dwierenga Excellent idea. Could I kindly ask you to open an issue here and we will follow-up.

We are working on setting up centralised information and documentation on the KornShell and this definitively belongs there :-)

[nickpapadonis]

I have to search my messages. In the meantime, someone pointed me to a page in the KSH book by David Korn that documented this issue as a method to add control characters. I then saw this and realized if that book is taken as the standard, this is not a bug.

Yikes!

On May 4, 2020, at 2:48 PM, Dan Wierenga notifications@github.com wrote:

oh, you are right regarding also happens in vi mode! it never happened to me since I have added emacs-like cursor movement keybindings to my vi mode so that I can use ^B and ^F from within vi insert mode. and in this combo \^B actually just backspaces :).

Sorry to comment on a closed issue, but is there any chance you could share those keybindings? Maybe start a wiki page in this project? There's a general lack of information around the web for the esoteric way to get KSH to interpret key strokes.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ksh-community/ksh/issues/8#issuecomment-623639316, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD272XVWR6TTNQXIG5J5HMLRP4EXHANCNFSM4LS2ANNQ.

[nickpapadonis]

Read through this. The relevant documentation in the KSH book describing this “feature” is pictured.

Personally, I think the feature is confusing, but I can also see use for it. It would be interesting to find out if anyone is using KSH this way.

https://github.com/ksh-community/ksh/issues/8#issuecomment-623639316 https://github.com/ksh-community/ksh/issues/8#issuecomment-623639316

On May 4, 2020, at 2:48 PM, Dan Wierenga notifications@github.com wrote:

oh, you are right regarding also happens in vi mode! it never happened to me since I have added emacs-like cursor movement keybindings to my vi mode so that I can use ^B and ^F from within vi insert mode. and in this combo \^B actually just backspaces :).

Sorry to comment on a closed issue, but is there any chance you could share those keybindings? Maybe start a wiki page in this project? There's a general lack of information around the web for the esoteric way to get KSH to interpret key strokes.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ksh-community/ksh/issues/8#issuecomment-623639316, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD272XVWR6TTNQXIG5J5HMLRP4EXHANCNFSM4LS2ANNQ.

[McDutchie]

It's worth pointing out here that ksh's "raw" POSIX (neither vi nor emacs) editing mode is implemented in two different ways, depending on whether a UTF-8/multibyte locale is active or not. By default, raw POSIX editing mode is handled in edit/edit.c. But in UTF-8 locales, control is passed to edit/vi.c.

And vi.c contains a number of conditionals that are supposed to disable vi features if the -o vi shell option is not set. Unfortunately that is very buggy, so you get e.g. extremely broken tab completion (which is already fixed in 93u+m).

This also explains the bugs @saper made videos of (see above); evidently, a UTF-8 locale was active. If you set LANG=C or LC_ALL=C you'll see those bugs disappear.

[nickpapadonis]

So does that mean this is a bug or intended behavior?

On Jul 2, 2020, at 10:12, Martijn Dekker notifications@github.com wrote:

 It's worth pointing out here that ksh's "raw" POSIX (neither vi nor emacs) editing mode is implemented in two different ways, depending on whether a UTF-8/multibyte locale is active or not. By default, raw POSIX editing mode is handled in edit/edit.c. But in UTF-8 locales, control is passed to edit/vi.c.

And vi.c contains a number of conditionals that are supposed to disable vi features if the -o vi shell option is not set. Unfortunately that is very buggy, so you get e.g. extremely broken tab completion (which is already fixed in 93u+m).

This also explains the bugs @saper made videos of (see above); evidently, a UTF-8 locale was active. If you set LANG=C or LC_ALL=C you'll see those bugs disappear.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[McDutchie]

What is shown in this comment is a bug: https://github.com/ksh-community/ksh/issues/8#issuecomment-605896180

What is shown in this comment is part intended, part bug: https://github.com/ksh-community/ksh/issues/8#issuecomment-605903944 The ^? is a bug, but it's not a bug that Shift+C puts you in a separate buffer that you can't backslash past until you escape out of it again; that is standard vi behaviour (though vim acts differently).