auto scaling group health check grace period

[emilburzo]

Hi!

I've noticed that the health check grace period for boxfuse's auto scaling groups is set to 300 seconds.

However, in our case, it only takes about 1-2 minutes from the time the auto scaling group is created, to the time the ELB puts the instances into service.

I thought maybe the instances are put into service earlier if they already respond to the ELB healthcheck, but the AWS docs mention:

While the EC2 status checks and ELB health checks can complete before the health check grace period expires, Auto Scaling does not act on them until the health check grace period expires.

so that shouldn't be the case.

Any idea why they are put into service so early?

Thanks!

[axelfontaine]

I don't have a good answer for you here. I am afraid this is more a question for AWS support as to why their docs don't match reality.

[emilburzo]

@axelfontaine sure, that makes sense.

I just wanted to check first that boxfuse doesn't set it to zero (or similar) first and then increases it to 300.

Is the value user-configurable in any way?

[axelfontaine]

Yes, it is configured using the healthcheck.timeout property. See https://boxfuse.com/docs/commandline/run#healthcheck.timeout

[emilburzo]

For anyone else that might stumble upon this, AWS support answer:

I understand that you have an ASG which has the health check type as ELB and grace period as 300 seconds but the instance is put into service by ELB even before the grace period completes. Please correct me if I am wrong.

I'd like to inform you that ELB health checks are a bit differnt from ASG health checks. ELB will start doing health checks as soon as the instance is up and running. It won't wait for the ASG health check grace period. In case ELB health checks pass, it will put the status in service on the ELB console without waiting for health check grace period.

During a health check grace period, only the ASG does not consider the health check values of an instance and take no scaling action on it. In a nut shell, a health check grace period simply means how long Auto Scaling does not health checks an instance and do not take any action on it. In that period ELB can mark the instance in service as it's health checks are passing and instance will start serving traffic.

Health grace period is to give your instance ample warm-up time so that ASG shall not terminate it.

Moreover, the line "While the EC2 status checks and ELB health checks can complete before the health check grace period expires, Auto Scaling does not act on them until the health check grace period expires." does not mean that there will be no health checks and instance won't be marked in service, rather it means that ASG will not terminate the instance/take any scaling action even when the health checks fail during the health check grace period.