search

awslabs / amazon-ebs-autoscale

Don't run out of disk space on your EC2 instance when generating or working with large files. Automatically add EBS volumes to a filesystem mount point in response to disk utilization.
MIT License
113 stars 60 forks source link

Please Update latest to include commit f64f64e which updates ebs-autoscale #37

Closed pptb-aws closed 2 years ago

pptb-aws commented 2 years ago

When running the default user data template and RunInstance command, I noted that, in larger tests some later parts of added user data scripts were failing to run. Looking at the system logs I saw this error,

[   27.503148] cloud-init[8419]: An error occurred (RequestLimitExceeded) when calling the DescribeVolumes operation (reached max retries: 2): Request limit exceeded.

Investigating CloudTrail I saw that each Instance was making a DescribeVolumes call every second. Digging into the code I found that bin/ebs-autoscale was the only place that runs describe-volumes. I confirmed this by adding the below log line, "running describe-volumes" in the get_num_devices definition and I see it printing out every second.

  for i in $(eval echo "{0..$max_attempts}") ; do
      local attached_volumes_response=""
      logthis "running describe-volumes"
      attached_volumes_response=$(
        aws ec2 describe-volumes \

This has been fixed in the mainline version by moving the NUM_DEVICES=$(get_num_devices) call into the if statement that only runs when the storage is running low.

Given that the default instructions have people running from the latest tag I believe at least this change should be pushed out ASAP.

golharam commented 2 years ago

This would also explain that when running top it seems that calls to aws clicare constantly happening.