Support wait command for SSM agent to become available on new instance

[glucas]

We have a use case where we launch an EC2 instance and when it is available send it an SSM command. There does not appear to be a way to wait for the SSM agent on the instance to be up without polling the ssm instance information, e.g.

until aws ssm describe-instance-information --instance-information-filter-list "key=InstanceIds,valueSet=${someId}" | grep -q 'AgentVersion';
do
   sleep 15;
done

It would be great if wait were supported as it is for EC2 instance state.

[justnance]

@glucas - Thanks for you post and recommending a waiter for SSM. I have marked this issue as a feature request.

[bmckinle]

Any progress on this feature request?

[dsounded]

+1

[agimenez]

+1 This is needed if you want to integrate any kind of automation (CI/CD) with SSM commands.

[sevontheweb]

+1 It would be even better to integrate a cloudwatch event when the SSM agent is ready. In this case a Lambda could for example be triggered and execute a command on the instance. At the moment using the EC2 state event (ec2=running) is problematic because when the event is triggered there is an arbitrary time until an SSM command can be sent to the instance (even though the state of the instance =running)

[vadimkantorov]

Also stumbled on this

[tim-finnigan]

Hi everyone, thanks for your patience regarding this request. I just reached out to the SSM team for feedback. I will update this issue as soon as I hear back.

[tim-finnigan]

P54243832

[tim-finnigan]

Thanks again everyone for your patience regarding this. I heard back from someone internally who said:

If the use case is to run the command when an instance first checks in with the SSM service, you can use SSM State Manager and create Associations without any scheduler to bootstrap the EC2 instances with the runCommand, which will issue the command to the EC2 managed instance automatically.

Since the association also triggers when the instance goes offline and checks-in again, you can make your command based script idempotent, which will create a file at a location for example /executedonce and the script at the start should check if this file exists or not. Based on the results, the script can proceed or exit.

The EC2 waiters are for the instances which are launched by the EC2 and are likely to change the state. Here the dependency would be on the EC2 and SSM waiter even will be making the same API calls at the backend to find the result. The SSM team may not be likely to consider this feature request when we have State Manager already to address the use-case.

I hope that addresses this issue. If anyone wants to clarify anything or provide more feedback on a use case please do and we can revisit this.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[paulhopkins11]

@tim-finnigan could you please clarify this as I am not seeing the behaviour you describe. I have a State Manager Association with no schedule. This is using a resource group to select targets. I see the association ran when first created (as expected) If I stop and then start an instance in that target group I do not see that association being executed again. Should this work?

[shepherdjerred]

@tim-finnigan could you please clarify this as I am not seeing the behaviour you describe. I have a State Manager Association with no schedule. This is using a resource group to select targets. I see the association ran when first created (as expected) If I stop and then start an instance in that target group I do not see that association being executed again. Should this work?

If you don't provide a schedule then your association will execute only once.

[Regancm]

SSM Associations are a terrible way to bootstrap instances.

After it runs it sits in the state forever and clutters up the view of recurring associations.

Automating association creates then deleting them later is fundamentally at odds with most common provisioning patterns and tools. Take cloud formation for example, you would need to run CF to provision the ec2 instance then add an association to bootstrap it. There is no way to have the association clean up, you have the remove the association resource from CF then deploy a stack update.

Yes, I know CF itself has much better tools for bootstrapping, this is just an example of the pattern suggested above by @tim-finnigan. However I believe SSM Documents are way more powerful than CF init scripts. Unfortunately windows servers exist and require handling reboots and numerous other tasks that are cumbersome and clunky in CF.

I would love to see AWS make SSM a first class citizen in ec2 service. Imagine just simply being able to point a launch at an SSM Document and behave the same way as Userdata where it runs once or can optionally run every time the instance is restarted. That would become a beloved feature very quickly.

Currently if you wanted to run ssm document to bootstrap an EC2 instance you can run one from userdata scripts BUT you first have to wait for the ssm agent to come online and that doesn't always happen smoothly so you have to add multiple retries and helpers. It's a total mess and requires giving the EC2 instance WAY too many IAM permissions just to simply see the status of itself to SSM since the agent status is not available unless you also write more code to snoop around in the local SSM Agent Logs.

It makes no sense that AWS created SSM and it doesn't integrate well with EC2 and other provisioning services or deployment tools.