Session freezes when trying to exit after running Terraform commands using the CLI

[juanma-cvega]

Hi,

I'm using SSM Session Manager through the CLI to connect to a bastion host I issue Terraform commands from. Every time after I run a terraform init command, I'm unable to exit the session, the console just freezes after executing exit and I have to either kill the session manually or wait until it expires. If I exit the session after connecting to the instance or after running git commands for example there is no problem.

Environment: Bastion: EC2 instance using a golden image built with AWS Image Builder from the latest Amazon Linux 2 AMI and with the latest SSM agent version. Rebuilt every week. Local: MacBook Pro. macOS Big Sur. SSM Manager Plugin version: 1.2.205.0 AWS CLI version: 2.2.18

I tried tailing the logs from a session opened through the AWS console while executing the exit command on a terminal through the SSM Manager plugin. When I exit the session without executing a Terraform command, I can see a bunch of logs appearing. Doing the same thing after executing a Terraform command, I see nothing in the logs. I execute the CLI command to connect to the instance without any document associated.

[juanma-cvega]

To follow up on this, I've been able to narrow down the cause a bit. It seems it's related to Terraform downloading modules from Git sources. In my case, I have some modules I import from a private BitBucket repository using tags. Whenever I execute a Terraform command that downloads from these sources and that Terraform prompts for the username/password, it freezes after exiting the terminal. However, if I execute a Git command prior to the Terraform one that asks for the username/password which results in these being cached so that Terraform doesn't need to ask for them again, then it exits successfully.

[yuting-fan]

Hi juanma-cvega@,

Thanks for reporting the issue. Here're a few follow-up questions in order for us to reproduce and look into your issue further:

In your case where the exit command hung, was there any output for terraform init like Initializing modules...Downloading git, or was there no output at all while the session hung?

Was the session hang after you type in username/password then exit, or when you directly typed exit via keyboard input while Terraform prompted for username/password?

Which session type were you trying to use (Standard_Stream, InteractiveCommands, NonInteractiveCommands or Port)?

Thanks, Yuting

[juanma-cvega]

Hi, sorry for the late reply. To answer your questions:

• When I run the Terraform scripts, they work perfectly. They download modules and apply changes without any issues and I can see the logs from Terraform like normal. It's when I want to close the session with the exit command that it just hangs if I previously run a Terraform script that asked me for the username/password.
• The session hangs if I introduce my username/password after Terraform asks for them. If I exit without entering them, it doesn't hang when closing the session.
• I didn't know there were different type of sessions. I tried googling those but I couldn't find them. I just use the command aws ssm start-session --region eu-west-1 --target <<instance id>> to connect. Is there any doc where these session types are described?

I just realised there is some extra information I didn't provide. I have set the system up so instances are created on demand and destroyed after the session is closed. I have created some predefined scripts using the EC2 user data field that reads the username/password from a secret and uses them to clone a git repo. At this point, it's the root user that clones the repo and then I change the permissions. When I connect to the instance, I can navigate to the repository and execute a Terraform command without having to first clone the repo (that's why I can execute a Terraform command over a Git repo without actually running a git clone command first). I still need to use my credentials to download the Terraform modules stored in a Git repository when I do it this way as git hasn't stored them yet for the ssm user, which causes the terminal to hang anytime I execute the exit command afterwards. If instead of directly execute the terraform init command to download the modules I run a Git command that prompts me to enter my credentials (like git pull for instance), then the Terraform command can download the modules without having to type the credentials again and in this case it works fine. This is the script I use in the user data field:

echo "Configuring git credentials..."

cat >> git-askpass-helper.sh << EOF
#!/bin/sh
exec echo "${PASSWORD}"
EOF

chown ssm-user:ssm-user git-askpass-helper.sh
chmod 500 git-askpass-helper.sh
export GIT_ASKPASS=/home/ssm-user/git-askpass-helper.sh

git config --system credential.helper 'cache --timeout 28800'
git config --system user.name ${USERNAME}
git config --system user.password ${PASSWORD}
echo "Git credentials configured"

echo "Cloning repositories..."

git clone myrepo1 &
git clone myrepo2 &
wait

chown -R ssm-user:ssm-user myrepo1
chown -R ssm-user:ssm-user myrepo2