TargetNotConnectedException prevents us from running important commands in our application

[george-silva]

Hello!

We have an application with multiple services. All of our services (and backend workers, etc), have exec: true in our manifests.

Since last night, we deployed new things on two environments. These environments right now cannot run commands anymore: copilot svc exec --env <env> --command "python" (or whatever command).

This is what it is returning:

Service basalt found in environment staging
Execute `python` in container basalt in task <my-task-id>.
✘ Failed to execute command python. Is `exec: true` set in your manifest?
✘ execute command python in container basalt: execute command: TargetNotConnectedException: The execute command failed due to an internal error. Try again later.

We even deleted an entire environments and redeployed, but no dice.

This prevents us from running migrations in our database (part of our deploy pipeline).

Any help?

[george-silva]

Having AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as variables in COPILOT will forbid you to connect to your ECS tasks.

Renamed the variables and and it all worked. Can we add some disclaimer to the docs?

https://github.com/aws-containers/amazon-ecs-exec-checker/issues/49

[efekarakus]

wow great find @george-silva!

Possible explanation

I didn't know either that setting the AWS SDK env variables could cause an issue 🙏 , in hindsight it makes sense although it's not obvious 😅 :

• ECS requires the following permissions in the task role when using exec which copilot adds by default:

  {
  "Version": "2012-10-17",
  "Statement": [
     {
     "Effect": "Allow",
     "Action": [
          "ssmmessages:CreateControlChannel",
          "ssmmessages:CreateDataChannel",
          "ssmmessages:OpenControlChannel",
          "ssmmessages:OpenDataChannel"
     ],
    "Resource": "*"
    }
   ]
  }

  https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html#ecs-exec-enabling-and-using

• When passing down the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to the container, I believe they take precedence over the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI when fetching credentials: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html. If your environment variable credentials don't have the ssmmessages:: permissions then an error occurs.

Mitigation

This is a tough spot, we could error if users specify in the manifest the AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY but it might be limiting users for unclear reasons.

We could print a warning during svc package but it's easily to miss that too, like documentation updates 💭

[george-silva]

If you have the key/secret secrets and exec:true configured in your manifest, it would be superb if Copilot gave out a big warning saying that exec might not work correctly.

A note in the docs with the fix (adding the right permissions to your AWS_ACCESS_KEY_ID/SECRET user or changing the variable names) would go a loooong way.

Thanks for the reply!

[cristobalmackenzie]

We're having the same issue. It started happening a few minutes ago after I accepted a SessionManager plugin update.

$ copilot svc exec                                                                                                                                                                                                         [17:51:35]
Looks like the Session Manager plugin is using version 1.2.295.0.
Would you like to update it to the latest version 1.2.312.0? Yes
Archive:  /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle.zip
   creating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/
  inflating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/install
  inflating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/THIRD-PARTY
  inflating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/seelog.xml.template
  inflating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/LICENSE
   creating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/bin/
  inflating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/bin/session-manager-plugin
  inflating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/NOTICE
  inflating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/README.md
  inflating: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/RELEASENOTES.md
 extracting: /var/folders/fy/46wsqgfs0n1cjt2827mv5g780000gn/T/ssmplugin3193845298/sessionmanager-bundle/VERSION
Password:
Symlink already exists. Removing symlink from /usr/local/bin/session-manager-plugin
Creating Symlink from /usr/local/sessionmanagerplugin/bin/session-manager-plugin to /usr/local/bin/session-manager-plugin
Installation successful!
Service: django
Execute `/bin/sh` in container django in task 0c64cd3ab9674594a0257505100617c0.
✘ Failed to execute command /bin/sh. Is `exec: true` set in your manifest?
✘ execute command /bin/sh in container django: execute command: TargetNotConnectedException: The execute command failed due to an internal error. Try again later.

It might be the case that lots of folks are going to start cropping up with the same issue.

Thanks @george-silva for the research !