ECS Agent Optimization : Reduce CPU usage in event of multiple ecs tasks placements

[FLavalliere]

Summary

When you have an environment where you have 60+ containers per nodes ( let say 2 nodes with 120 tasks). And let say you have a t3.2xlarge (or t3.large). If they died (force shutdown). New nodes have high CPU due to too many concurrent dockerd process going on..

Description

If you need to replace a node (due to an unexpected failure) or replacement. Let say you had 2 more nodes Available that were empty. The new nodes are just maxing out on CPU / Memory (during docker untar unpigz.)

Is there an option to limit how many concurrent 'Docker creation process' is being launched?

Ie: On a 8CPU we might want to limit to 4 Docker Pull/Extract/Up at a time to ensure not to impact other Docker containers.

If such feature doesn't exists, could we get a Ecs-Agent Configuration to tell how much concurrent maxinum is allowed on a node? and default would be (number of cpu's? or maybe 3/4 of the total available CPU's)

[sparrc]

There is not an option for the ecs agent to do that.

It sounds like you would be better served by having more nodes with fewer tasks per node. Is there a reason you are unable to architect your system that way?

[FLavalliere]

Indeed, with many many more nodes and higher CPU/Memory reservations that would not be an issue. Though this is a cost constraint, for dev/qa environments. And these systems with over 80 containers (consume verry minimal cpu / memory reessources).

Since its mostly using spot instances its not uncommon that some EC2 might get replaced. So the only issue is when there are a node that is dieing abruptively / draining the 80 tasks. That adds a lot of 'CPU / Memory Ressource usage' on the other nodes affecting performance / potential healthchecks.

Even if the tasks are highly available it can impact tasks on the other nodes.

So it would be nice i think to have a features that we can toggle On to allow only X containers at the same time to be scheduled / deployed else the other ones would be delayed / cancelled.

I believe its a simple feature to implement, i havent checked the code but having a global counter could help in scenario where there are a lot of small containers with small resources footprint.

[nathanpeck]

If I understand correctly you are saying that these 60 containers are stable on the available CPU capacity once they have finished starting up, however you are seeing instability if all 60 are trying to startup at the same time? So you just want to slow down the speed at which these containers are started, while still being able to eventually launch 60 containers?

Unfortunately ECS does not have a concept of slowing down the task launch like that. One thing you could try doing is using the task healthcheck grace period to give these tasks more grace period while starting up before healthchecks come into effect. That way even if they are burning lots of CPU while starting up and failing healthchecks ECS will allow them to fail healthchecks during the grace period until they eventually stabilize.

[FLavalliere]

Yes that is correct. The system is stable with these many containers.

The issue is really that ECS try to schedule all of them and the ECS-Agent takes that and simply launch them on the EC2.

The Docker / Untar of the various docker images takes a lot of CPU. And i was wondering if there could be an easy flag to maybe limit the number of concurrent docker pull type of thing.

So currently it eventually stabilize. But the VM CPU's are so high that it impact other Services. In some case it takes a lot more memory (thanks for a small swap here) and for the ECS-Agent OS Reserved Memory option that I granted more Space to allow this kind of spike in memory usage.

Though if we could rate-limit / or exponential backoff at the 'ecs-agent' level that would be awesome. I understand though its not a problem many might face though.