hackoregon / civic-devops

Master collection point for issues, procedures, and code to manage the HackOregon Civic platform
MIT License
11 stars 4 forks source link

Add ECS parameters to reduce the cleanup time/behavior. #163

Closed iant01 closed 6 years ago

iant01 commented 6 years ago

The default ECS agent period to wait until cleaning up of stopped/dead containers or replaced task definitions is three hours. When services are thrashing due to a bad container or health check endpoint, this can eat up a lot of space on the docker volume. Putting the configuration parameters into the cloudformation template enables changing the behavior without having to SSH into the instances, make the changes on both, remember to stop/start the ecs agent (and possibly docker as well)

iant01 commented 6 years ago

These changes will cause ecs agent to look every 10 minutes for container images that have died or been stopped that are older than 15 minutes in time. To limit the amount of time spent on cleanup, a max of 10 images will be deleted per cycle. Task information will be cleaned up 5 minutes after the task has been information has been updated. echo "ECS_IMAGE_CLEANUP_INTERVAL=10m" >> /etc/ecs/ecs.config echo "ECS_IMAGE_MINIMUM_CLEANUP_AGE=15m" >> /etc/ecs/ecs.config echo "ECS_NUM_IMAGES_DELETE_PER_CYCLE=10" >> /etc/ecs/ecs.config echo "ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=5m" >> /etc/ecs/ecs.config

iant01 commented 6 years ago

merge/pull request 36 submitted.

iant01 commented 6 years ago

closing, resolved with PR merge