Open alxsbn opened 3 years ago
Thanks for reaching out to us!
Could you please share logs for Session Manager plugin for us to investigate this further? More information on logging can be found here - https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-configure-logs.
Did you get any resolution on this?
I am also facing the same issue when I tried to login via Jenkins job. while using the command line it's working fine.
I had similar issue with "aws ecs execute-command ..." and it seems it's related to tty being not available when running in ci
I fixed it with "unbuffer aws ecs execute-command ..." to run it with tty available
It's linked to tty. I bypass the problem with ssm send-command with remote script execution
@alxsbn Can you explain more what you meant by "ssm send-command with remote script execution"? We're running into the same issue with
aws ecs execute-command
I don't know if this makes a difference, but we're using ECS with Fargate, not EC2.
@emmahsax Our first reflex was to run a ssh command, then the others commands we need (for us it was linked to CD, so rsync a repo then restart a pm2 service). Since we have this tty problem we just write a shell script with our commands an fire him with ssm send-command
I have the same problem, I followed https://docs.aws.amazon.com/systems-manager/latest/userguide/session-preferences-shell-config.html to add bash -i
to my Linux Shell Profile
I enabled logging and see:
2021-05-11 11:10:09 DEBUG Start File Watcher On: /usr/local/sessionmanagerplugin/seelog.xml
2021-05-11 11:10:09 DEBUG Start Watcher on directory: /usr/local/sessionmanagerplugin
2021-05-11 11:10:09 DEBUG Calling Initialize Datachannel for role: publish_subscribe
2021-05-11 11:10:09 INFO Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/botocore-session-1620726103-0af510cb5dd7ad804?role=publish_subscribe)
2021-05-11 11:10:09 INFO Successfully opened websocket connection to: %!(EXTRA string=wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/botocore-session-1620726103-0af510cb5dd7ad804?role=publish_subscribe)
2021-05-11 11:10:09 INFO Sending token through data channel wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/botocore-session-1620726103-0af510cb5dd7ad804?role=publish_subscribe to acknowledge connection
2021-05-11 11:10:09 DEBUG WebsocketChannel: Send ping. Message.
2021-05-11 11:10:10 DEBUG Processing stream data message of type: output_stream_data
and the logs end there.
I saw the same issue when running aws ssm start-session --target INSTANCE_ID --document-name AWS-StartInteractiveCommand --parameters command="bash -i"
:
Starting session with SessionId: botocore-session-1620726103-06d6cbb9a7fcd54a6
ssm-user@ip-10-129-212-60:/var/snap/amazon-ssm-agent/3552$ Cannot perform start session: EOF
It also messes up the terminal, I have to reset
to get what I type to show properly again after it exits.
I'm using version 1.2.54.0
Session manager plugin has been designed to be invoked from a terminal for shell scenarios so it can interpret control sequences returned by pseudo terminal and display appropriately. When a shell session is started, session manager plugin also fetches terminal size on client machine and sends it to pseudo terminal on target server. We suspect one of these actions might be failing for you. Please let us know if you are invoking session manager plugin from a terminal and still seeing error of Cannot perform start session: EOF
.
Yes, I am invoking it from a terminal. I haven't changed anything about the instances I'm connecting to, they're launched from an AMI I baked weeks ago. It's working today, without changing anything in my local environment (same version of session-manager-plugin
, same AMI/instances)
When a shell session is started, session manager plugin also fetches terminal size on client machine and sends it to pseudo terminal on target server.
ah, I think I get it. When I try the same command from a terminal with an enormous scrollback, it fails with Cannot perform start session: EOF
. Using a new terminal session clears the scrollback, and aws ssm start-session
works as expected.
Thanks for letting us know that you were able to get past this issue. Please let us know in case of further issues.
I had similar issue with "aws ecs execute-command ..." and it seems it's related to tty being not available when running in ci
I fixed it with "unbuffer aws ecs execute-command ..." to run it with tty available
@olfway You saved my day! I had the exact same issue and sudo apt-get install expect; aws ecs execute-command ....
worked like a charm! Thanks.
More on unbuffer
& expect
https://stackoverflow.com/questions/66066753/aws-start-session-end-with-cannot-perform-start-session-eof
Side note, you can run single commands like so
unbuffer aws ssm start-session \
--document-name 'AWS-StartNonInteractiveCommand' \
--parameters '{"command": ["sudo sh -c \"systemctl status boot_scripts && ( journalctl -u boot_scripts | cat )\""]}' \
--target "$instanceid"
maybe this command can help
name: Install AWS CLI run: | sudo apt-get update sudo apt-get install -y awscli aws --version
name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v1 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
name: Install Session Manager Plugin run: | curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb" -o "session-manager-plugin.deb" sudo dpkg -i session-manager-plugin.deb sudo apt-get update sudo apt-get install -y session-manager-plugin sudo apt-get install -y expect session-manager-plugin --version
name: Create Start SSH Session shell : sh run: | aws ssm start-session \ --target ${{ secrets.INSTANCE_ID }} \ --document-name AWS-StartSSHSession
Hello, any update about this?
@baguilarq, please read the entire thread and links before commenting. This ticket should be closed. I have posted a solution for this problem above.
More on
unbuffer
&expect
https://stackoverflow.com/questions/66066753/aws-start-session-end-with-cannot-perform-start-session-eofSide note, you can run single commands like so
unbuffer aws ssm start-session \ --document-name 'AWS-StartNonInteractiveCommand' \ --parameters '{"command": ["sudo sh -c \"systemctl status boot_scripts && ( journalctl -u boot_scripts | cat )\""]}' \ --target "$instanceid"
Can you elaborate how is this solve the issue? It doesn't for me and I have both unbuffer
and expect
available.
I can log in to most of the instances expect some with the error above and I can't grasp the difference.
More on
unbuffer
&expect
https://stackoverflow.com/questions/66066753/aws-start-session-end-with-cannot-perform-start-session-eof Side note, you can run single commands like sounbuffer aws ssm start-session \ --document-name 'AWS-StartNonInteractiveCommand' \ --parameters '{"command": ["sudo sh -c \"systemctl status boot_scripts && ( journalctl -u boot_scripts | cat )\""]}' \ --target "$instanceid"
Can you elaborate how is this solve the issue? It doesn't for me and I have both
unbuffer
andexpect
available. I can log in to most of the instances expect some with the error above and I can't grasp the difference.
did you try to do that without break lines?
Thank you. The command that I tried to execute is what OP tried: aws ssm start-session --target i-xyz
.
I added unbuffer
to the beginning, same error: unbuffer aws ssm start-session --target i-xyz
. My question was, why using unbuffer
would make it work?
This question should be closed, as I've answered it already above. Google and StackOverflow can help you with why a solution or command works. I recommend the spirit of exploration always. That being said, I feel nice, in the hope that no one else has questions about this.
Historically, the command aws ssm start-session
is designed to start a direct connection to the target instance by means of a terminal (sh/bash/zsh/etc). Because of this aws ssm start-session
requires a TTY to be available, which is not in CI/CD pipelines. The tty command
of the terminal basically prints the file name of the terminal connected to standard input.
Thus aws ssm start-session
is requesting access to standard input and does not have that keyboard access in a CI/CD pipeline context. We can mock this by using unbuffer
. If a keyboard, or stdin, is actually needed to input real-time for a program, you can use the expect
command, which is commonly used with unbuffer
. I do not recommend this, however, as 99.9999% of the time, the program's built-in command line arguments will suffice.
I try to instanciate a vanilla
aws ssm start-session
on a Github actions Jobs (without SSH).The session is well instanciated (i can see it on the AWS Session Manager dashboard) but always finish with a Cannot perform start session: EOF
FYI I can instanciated a sesion outside of GH actions (.e.g. on my laptop) with the same account / role / policy / instances (ami-0ea4a063871686f37).
Here's my job YAML :
The results :
Here are the CLI tools packaged with Ubuntu-latest virtual env (20.04.2 LTS) :