Targeting all instances does not include hybrid-joined managed instance (mi-...)

[IgnacioKaplan]

After doing the automated deployment from AWS SSM, following the aws distributor official package, it all went well, the aws role and the state manager association was defined as expected. It actually successfully deployed the falcon sensor to some of the instances that were missing it.

However we noticed it is only targeting local instances (i-...), and we do have a number of instances hybrid joined (mi-...) in our aws-SSM.

If we compare with similar associations that we run, when we targeted all the instances, the hybrid managed instances are included. So we suspect there is something on the automation document causing this.

As alternative to use the all Instances wildcard ["*"} we preloaded a variable with all the instances list and passed the list of instances:

data "aws_ssm_instances" "online" { filter { name = "PingStatus" values = ["Online"] } }

However using this approach, it is limited to 50 instances and there is no way around it that we could find. Error: │ │ with aws_ssm_association.association, │ on main.tf line 64, in resource "aws_ssm_association" "association": │ 64: values = data.aws_ssm_instances.online.ids │ │ Attribute targets.0.values supports 50 item maximum, but config has 205

Have you find the same problem or what would be the best approach to make sure all the SSM instances are included (local and hybrid joined)?

many thanks, Ignacio

[ffalor]

Hey can you give an example of an automation document you use that is able to target hybrid managed instances?

[IgnacioKaplan]

To be honest after having a closer look to it, we have few of the state manager associations that are targeting hybrid instances, but in all these cases they are using a command document, instead of an automation document (I think previously named as runbooks). Does it mean there is an AWS limitation where cannot be target hybrid instances with automation documents?

[IgnacioKaplan]

Hi again, additionally to my previous comment, I have been doing some tests by targeting specific instances instead of all of them.

I proved that I can manually assign a hybrid instance (mi-..) from the console and from code. And it works as expected. The problem now is that using this alternative approach, I need to do a pre-selection of the instances but the parameter/Key InstaceIDs must have length less than or equal to 512, so I can't pass all of them.

If I do this manually from the aws console, I can put on the InstancesID field as many instancesIds as I want without issue.

So at this stage I still have 2 issues that cannot be resolved: -> If I target all the instances, only local instances are taken into account -> If I pass a string with all the instancesIDs that I want to target, is not an option giving the parameter limitation from code.

Attached screenshot

[ffalor]

Does it mean there is an AWS limitation where cannot be target hybrid instances with automation documents?

That's my concern I haven't been able to find anything that says automation documents can target hybrid instances. Is this something you'd be willing to ask AWS about?

If I pass a string with all the instancesIDs that I want to target, is not an option giving the parameter limitation from code.

The 512 limit is from AWS not our document. When I use the console and create an association and pass in a list of instanceids I get for Key InstanceIds must have size less than or equal to 25 It looks like there is a 25 instance limit. Where were you able to pass in as many as you want?

/edit it looks like at 25 instances technically you are at 500 characters and when you add in mi instances you can get over 512 so the two limits look to be the same maybe?

[IgnacioKaplan]

I didn't hit that limitation when doing the setup manually. I added 46 instances.

This is how the setup looks like when I do it manually:

[IgnacioKaplan]

I will try raise with AWS the question around why when targeting all the instances with "*" only local instances are included.

Maybe worth to ask why we hit the 512 characters limitation only when deploying from code (in my case terraform) and from the aws cli.

[ffalor]

Please do - I'll continue to look on my side.

On a separate note recently we released 1 click sensor deployment that checks which instances are missing the agent and uses SSM to add the sensor it may also be worth a check. It is integrated into the CrowdStrike console. https://falcon.crowdstrike.com/documentation/418/get-started-with-1-click-sensor-deployment

[IgnacioKaplan]

Yeah, I agree. I checked out that 1-click sensor deployment, and is quite nice how easy it is to use from the Falcon console to have full control over the deployments.

Now, when it comes to Falcon Cloud Security, things get a bit tricky. If we're using it in our setup, our parent company handles its management. They have a view of everything happening across all the companies, so connecting our AWS infrastructure directly doesn't seem like the right move. This is where I think the SSM deployment option makes more sense.

Choosing the SSM route seems like the logical call. It lets us keep our AWS setup separate and not tangled up with the parent company's operations. With AWS Systems Manager, we can take care of the automated deployments ourselves and still independent from the top level company operations.

[IgnacioKaplan]

The case is raised with AWS, I will update this chain if I get any valuable answer. For the moment we are looking into add a programmatic logic to split the managed instances in groups of 20 and create an association per group. Is not the best way forward but it can cover until this issue gets resolved

[IgnacioKaplan]

Hi, please see below the answer from AWS.

_During our call, we noticed that the association was only failing to target the hybrid instances when using an automation document; however, it was working for a command document. Upon observing this behavior, I reached out to the internal team since we couldn't find any documentation that clearly specifies this.

The internal team confirmed that indeed the above behavior is expected. If you wish to use an automation document, you will have to use tags because using the "All instances" option will not target all the managed instances. Having noted this, I attempted to raise a feature request, but I discovered that one was already pending. Unfortunately, there is no ETA available for when the feature will be available.

With the above tasks completed, we concluded our meetings and agreed that I can go ahead and close the case. If you encounter any other issues, please feel free to open another support case, and we will be delighted to assist.

Until then, stay safe and have a lovely day._

[IgnacioKaplan]

We have implemented a workaround using terraform that deploys one state manager association using the Crowdstrike document for every 20 instances under the SSM fleet manager. The only caveat is that we have multiple associations to cover all the managed instances but so far it works.

Thank you, Ignacio

[ffalor]

Hey @IgnacioKaplan I really appreciate you getting back to me on this. It's unfortunate that this limitation exists... I'm glad you found a workaround.

It sounds like, for those that are able to tag their instances they would only need 2 associations to hit every machine. One for all ec2 instances and one that targets a tag that exists on all hybrid (mi-) hosts.

I'll close this since this isn't something that is in our control. Feel free to reopen if something changes.