Problem interpreting signals emitted by terminal on BSD systems properly

AndrewFarley commented 2 years ago

NOTE: This issue's contents and title have been significantly changed as of Jan 15 because of some folks getting to the underlying issue.

The underlying issue is that when using a BSD-based system (like OS-X) this plugin sends some undesirable control signals. This appears to ONLY affect Macs, not Windows or Linux.

@pedros007 Believes this issue is related to the ControlSignals handled by a ShellSession

Reproduce steps which seems to only happen on a MacOS terminal:

Start a shell session aws ssm start-session --target i-00112233445566778
Once connected, press Ctrl-y
Expected behavior: paste from the kill-ring (which is empty, so nothing should happen)

Actual behavior: The session terminates with the log line

Cannot perform start session: read /dev/stdin: resource temporarily unavailable

The kludgy workaround currently is

Run stty dsusp undef on your host.
Start a shell session aws ssm start-session --target i-00112233445566778
Press Ctrl-y to paste from the kill ring (which is empty, so nothing happens). This is what every other OS does, and what Linux normally does when you SSH.

For more details about this issue please see some of the comments towards the bottom of the thread/issue.

justinmk3 commented 2 years ago

4. Only in JOE though, not in VI, or Nano.

Similer to https://github.com/aws/session-manager-plugin/issues/28#issuecomment-1001810688 , I think the problem is likely that Joe is having trouble with the $TERM and/or terminfo/termcap found on the remote machine and it's just not handling it as gracefully as Vi / Nano.

You can check this by trying the following:

note the value of $TERM in the remote session
on your local machine or any other machine with an equivalent joe build, invoke joe with that TERM value, e.g.: TERM=xterm joe and see if it behaves similarly.

AndrewFarley commented 2 years ago

@justinmk3 Confirmed this formatting issue happens with TERM=xterm via SSM, but it does not happen locally, nor via SSH. I set my local TERM to xterm, and set it after SSM-ed into the session. The formatting works perfectly via SSH and locally, but when connected via SSM it fails to format things, often eliminating prefix whitespace, occasionally deleting return characters as well so some text appears on the same line somehow.

Confirm this happens on both on Terminal and iTerm on OS-X. And this has happened as far as back as I can remember using SSM. Only gotten around to reporting it now because of the increasing usage and adoption.

Another issue I've found @justinmk3 is that copy/paste doesn't work via joe. When I use other editors it works fine, but using joe it doesn't send return characters. I do suspect you're spot on on that its related to term or termcap, but I don't know what other adjustments I can make to fix this. And, like I said, this exact setup works perfectly via SSH. Any other ideas to try?

AndrewFarley commented 2 years ago

Example:

# VIA SSH...
$ TERM=xterm joe test # Then CTRL-K + V to pagedown to the bottom and I see...
                                                <img src="./externals/Landingpage/images/googleplus.png" alt="" />
                                        </a>
                                </li>
                                <li>
                                        <a target="_blank" href="http://website.com">
                                                <img src="./externals/Landingpage/images/twitter.png" alt="" />
                                        </a>
                                </li>
                                <li>
                                        <a target="_blank" href="http://website.com">
                                                <img src="./externals/Landingpage/images/facebook.png" alt="" />
                                        </a>
                                </li>
                        </ul>
                </div>
        </div>
</div>

# VIA SSM...
$ TERM=xterm joe test # Then CTRL-K + V to pagedown to the bottom and I see...
                                                <img src="./externals/Landingpage/images/googleplus.png" alt="" />
                                        </a>
                                </li>
                                <li>
                                        <a target="_blank" href="http://website.com">
                                                <img src="./externals/Landingpage/images/twitter.png" alt="" />
                                        </a>
                                </li>
<li>
    <a target="_blank" href="http://website.com">
<img src="./externals/Landingpage/images/facebook.png" alt="" />
</a>
</li>
</ul>
</div>
</div>
</div>

Note the bottom-half of the lines above are displayed without their space prefixes.

justinmk3 commented 2 years ago

I'm not intimately familiar with SSM, but I think the next step would be to share more concrete details about your SSH config and how exactly you are invoking ssh, aws, and/or session-manager-plugin.

copy/paste doesn't work ... it doesn't send return characters ... Note the bottom-half of the lines above are displayed without their space prefixes.

Something in the stack (SSM, SSH, your terminal, ...) is eating control codes (cursor-positioning in this case). I would expect SSM to strictly pass-through the data so the terminal can handle it.

Depends on how SSH+SSM is invoked/configured...

AndrewFarley commented 2 years ago

@justinmk3 I don't really know how/what to look into to give more information on this. My setup is simply... OS-X Big Sur 11.6 iTerm or Terminal Install AWS CLI 2.0

# Then run
AWS_DEFAULT_PROFILE=myprofile aws ssm start-session --target i-0123456789

Then open up joe and can replicate these bugs, but only via SSM, not via SSH. I will try this on a freshly installed computer with none of my bash_profile or any other modifications to check if this is something that is specific to my computer's setup. I'll start there, but if you have any other ideas of how I can provide further feedback would be great.

My belief is still that, for the most part, the only issues I have are with using joe via ssm but no other editor has any issues like described above. This leads me to believe there's some minor incompatibility between this editor and SSM, and I do not know which one to blame or to try to fix. I'd be happy to try to dig in and fix this issue, once I identify which component it is.

justinmk3 commented 2 years ago

Thanks. I can reproduce the rendering issues (but not the exact CTRL-D behavior) by connecting that way. Forcing TERM=ansi at least fixes the rendering issues:

TERM=ansi joe

For whatever reason, ssm sends TERM=xterm-255color to the remote session. But I don't think that's the whole problem.
(Are you using screen or tmux, btw?)
Maybe the termios properties are not setup correctly (by SSM, e.g. here)?

In vim, I notice that CTRL-D, CTRL-W, and other control chars do not seem to reach vim in their "raw" form. So I don't think this is an editor-specific issue. Just happens that the symptoms are different amongst editors.

Conclusion

IIUC, ssm when invoked via aws ssm start-session --target ... does not use ssh. And it looks like InitDisplayMode is supposed to setup the terminal, but the linux impl does nothing.

For interactive sessions, that function may need to set some termios properties.

If I understand your conclusion and links, it seems like some setup in the init display mode for Unix-based sessions might need a few commands to help ensure all the control and cursor-related messages get sent/received in both directions. I have no knowledge at this specific area, but I could poke around and fiddle. And worst-case, maybe I/you/others could help refine what we've learned here into a very clear issue report for someone else to work out.

pedros007 commented 1 year ago

TL; DR: Run stty dsusp undef prior to starting an SSM session to allow Ctrl-y to properly yank from the kill ring when connecting to a Linux instance from my Mac laptop.

More details:

A coworker and I ran into an issue where we'd use SSM to ssh to an instance:

aws ssm start-session --document-name AWS-StartInteractiveCommand --parameters '{"command": ["sudo su ec2-user"]}' --target i-00112233445566778

but when we would yank from the kill ring (via Ctrl-Y), a few seconds later the session ends and the following is written to the terminal:

Cannot perform start session: read /dev/stdin: resource temporarily unavailable

I have only experienced this connecting from a Mac laptop to a Linux EC2 instance. Can't reproduce it when connecting from a Linux machine to a Linux EC2 instance. We noticed that on Mac stty -a shows that ^Y is mapped to dsusp. According to GNU docs on signal characters:

The DSUSP (suspend) character ... sends a SIGTSTP signal, like the SUSP character, but not right away—only when the program tries to read it as input

We found that SSM sessions will not terminate if you first unmap the dsusp signal. Here's how to do that:

stty dsusp undef

Is there some way that the session manager plugin can be updated to not fire the dsusp signal when running on a Mac and a user enters Ctrl-y?

pedros007 commented 1 year ago

On a BSD system like my Mac, perhaps the bytes emitted by SIGTSTP are different from the bytes sent by Linux, so the signal is never captured by the session manager plugin?

https://github.com/aws/session-manager-plugin/blob/c523002ee02c8b68983ad05042ed52c44d867952/src/sessionmanagerplugin/session/sessionutil/control_signals_unix.go#L25-L33

AndrewFarley commented 1 year ago

@pedros007 Man, this is exactly what I've been suffering from for ages. I'm glad someone dug harder than I did to figure this out. This alone is preventing me from completely eliminating my usage of SSH. My cli editor regularly uses CTRL-Y during editing and it disconnects me. I have to try really hard not to use a command sequence which is embedded deep in my unix-ey brain from 20+ years of using it. Every time it happens I curse this silly feature and its silly bugs.

I don't know enough about the internal unix control signals to recommend what the actual fix is, but I hope someone does and can make the fix to this plugin permanently so I can enjoy uninterrupted shell sessions. I'm pleased to here there's a command I can run stty dsusp undef to prevent this issue. Don't know how I'll remember to run it every time since SSM-ing doesn't really run as-user bash/profile stuff. But at least after I forget to do it once, this command will prevent it from me doing it twice in a row.

trharris78 commented 1 year ago

I put this command in my .zshrc on my mac so that the terminal where I run the commands is set up correctly, and that seems to do the trick. If you have any other problematic Ctrl+chars, see if they are mapped to something in the stty -a output. I think @pedros007 also had trouble with Ctrl+S in emacs and was able to fix it by unmapping that particular control character. Something like stty stop undef.

pedros007 commented 1 year ago

@AndrewFarley I wonder if this issue should be renamed. The current title implies it's a problem specific to your editor. However, it seems more like a problem interpreting signals emitted by the terminal on BSD systems. Perhaps the problem is related to the ControlSignals handled by a ShellSession? So maybe rename to something like Shell Session signal handling on BSD systems like MacOS?

Reproduce steps which seems to only happen on a MacOS terminal:

Start a shell session aws ssm start-session --target i-00112233445566778
Once connected, press Ctrl-y

Expected behavior: paste from the kill-ring (which is empty, so nothing should happen)

Actual behavior: The session terminates with the log line

Cannot perform start session: read /dev/stdin: resource temporarily unavailable

The kludgy workaround fix is to:

Run stty dsusp undef on your host.
Start a shell session aws ssm start-session --target i-00112233445566778
Press Ctrl-y to paste from the kill ring (which is empty, so nothing happens).

AndrewFarley commented 1 year ago

@pedros007 You're right, let me fix up the title and description of this now that we know what the issue is. Hopefully someone can fix it

jra commented 1 year ago

As an emacs user, this is killing me. I just switched from ssh to session manager (plugin). At least Ctrl-s and Ctrl-j are messed up. I think Ctrl-s just doesn't make it to emacs, and Ctrl-j comes through as something else, maybe Return? I am using Apple Terminal. I tried sty disuse undef; it did not help. I tried other terminals--iTerm2 and Alacritty--with MacOS; they didn't help with this situation.

trharris78 commented 1 year ago

@jra Try running stty -a and see what chars/signals Ctrl-S and Ctrl-J are mapped to. For example, my output:

speed 38400 baud; 61 rows; 272 columns;
lflags: icanon isig iexten echo echoe echok echoke -echonl echoctl
    -echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo
    -extproc
iflags: -istrip icrnl -inlcr -igncr ixon -ixoff ixany imaxbel iutf8
    -ignbrk brkint -inpck -ignpar -parmrk
oflags: opost onlcr -oxtabs -onocr -onlret
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow
    -dtrflow -mdmbuf
cchars: discard = ^O; dsusp = <undef>; eof = ^D; eol = <undef>;
    eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
    min = 1; quit = ^\; reprint = ^R; start = ^Q; status = ^T;
    stop = ^S; susp = ^Z; time = 0; werase = ^W;

My terminal has stop set to Ctrl-S, so I would type stty stop undef to remove that mapping. See what yours says and if Ctrl-J is mapped anywhere.

jra commented 1 year ago

@trharris78 Here is what I get from stty -a:

$ stty -a
speed 38400 baud; rows 73; columns 243; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke

I just tried stty stop undef and it seems to address the Ctrl-s problem. I don't see Ctrl-j in the stty -a output, so not sure what to do for that one.

trharris78 commented 1 year ago

Someone else having the same Ctrl-J problem here, no resolution

mikewyer commented 1 year ago

Sorry if this is a daft question: one of the responses to a related ticket mentioned that JOE puts the terminal in "raw" mode so it can handle ctrl-D itself. I don't see anywhere that the SSM client puts the local terminal in raw mode so it can transmit those control characters to the remote end. Would that be a reasonable fix?

jbbarth commented 1 year ago

Same problem on Ctrl+O with description in #79, I closed it as duplicate. The title of this issue should be updated to reflect that the problem is about handling of Ctrl+key inputs, it would be clearer.

jbbarth commented 1 year ago

I built upon the analysis from @justinmk3 (thanks!) and submitted a PR in #80. Everything is described in the PR description. It fixes the issue, but it needs some polishing and probably the eye of somebody experienced with such issues. It's great if some of you can confirm if the branch fixes the problem for you or not. 🙏

jamiegosling commented 9 months ago

Hi,

I can't comment on the PR as I don't have enough Go knowledge, but I have tested this and I can confirm this fixes the issue for me (specifically being able to use Ctrl+s on k9s, although other Ctrl key combinations seem to work as well) on Mac OS Sonama. Thanks @jbbarth!

AndrewFarley commented 9 months ago

@jbbarth Man, I LOVE you. You just made SSM actually usable for me. Using joe and performing CTRL-Y doesn't suddenly eject me from the SSM session. You single-handedly turned this from a tool I use to put an SSH key onto a machine, into something I can use daily. This works great for me, fixes all issues I highlighted in this issue. If you have some way to donate $$$ in some way (paypal/crypto/gift card) I would be happy to chuck you over a HUGE thanks.

Definitely hope some others can review this code and get this merged and released for everyone ASAP. I'm keen to get people I manage to be able to use this and completely remove SSH everywhere always.

yuting-fan commented 9 months ago

Thank you all for reporting the issue and the discussions, and @jbbarth for your pull request! Session Manager team has received this pull request, will review and merge the change in upcoming release.

d4rky-pl commented 1 month ago

@yuting-fan there's been two releases already since then, is there anything else you still need to be able to review and merge this?

aws / session-manager-plugin