Using lifecycled to handle Autoscaling terminations and spot terminations

[paulhamby]

Is using a single lifecycled daemon to listen and respond to Autoscaling terminations and Spot terminations a use case that is expected to be working?

I'm seeing some inconsistent results when attempting to use it in this fashion.

When I start the daemon with --no-spot it is able to handle the Autoscaling terminations just fine, so thanks! However, when I remove that flag and attempt to have it handle the Spot termination warning events as well, I'm not having success. Lifecycled does not pickup on the spot instance termination warning.

Additionally, when I restart the daemon (without changing configurations) sometimes I will see it logging that the spot listener is polling sqs for messages, which to me, is unexpected. Other times the spot listener will log that it is polling ec2 metadata for spot termination notices, which is more of what I expect. This happens on the same host, not changing configurations, it appears that just stopping and starting the daemon results in different behavior.

Example log excerpt with expected log entries:

time="2020-05-07T18:35:39Z" level=info msg="Looking up instance id from metadata service"
time="2020-05-07T18:35:39Z" level=info msg="Starting listener" instanceId=i-instanceid listener=spot
time="2020-05-07T18:35:39Z" level=info msg="Starting listener" instanceId=i-instanceid listener=autoscaling
time="2020-05-07T18:35:39Z" level=info msg="Waiting for termination notices" instanceId=i-instanceid
time="2020-05-07T18:35:39Z" level=debug msg="Creating sqs queue" instanceId=i-instanceid listener=autoscaling queue=lifecycled-i-instanceid
time="2020-05-07T18:35:39Z" level=debug msg="Subscribing queue to sns topic" instanceId=i-instanceid listener=autoscaling topic="arn:aws:sns:us-east-1:account:terminate-hook-name"
time="2020-05-07T18:35:40Z" level=debug msg="Polling sqs for messages" instanceId=i-instanceid listener=autoscaling queueURL="https://sqs.us-east-1.amazonaws.com/account/lifecycle
d-i-instanceid"
time="2020-05-07T18:35:44Z" level=debug msg="Polling ec2 metadata for spot termination notices" instanceId=i-instanceid listener=spot
time="2020-05-07T18:35:49Z" level=debug msg="Polling ec2 metadata for spot termination notices" instanceId=i-instanceid listener=spot
time="2020-05-07T18:35:54Z" level=debug msg="Polling ec2 metadata for spot termination notices" instanceId=i-instanceid listener=spot
time="2020-05-07T18:35:59Z" level=debug msg="Polling ec2 metadata for spot termination notices" instanceId=i-instanceid listener=spot
time="2020-05-07T18:36:00Z" level=debug msg="Polling sqs for messages" instanceId=i-instanceid listener=autoscaling queueURL="https://sqs.us-east-1.amazonaws.com/account/lifecycle
d-i-instanceid"

Example log excerpt with unexpected log entries:

time="2020-05-07T23:09:21Z" level=info msg="Looking up instance id from metadata service"
time="2020-05-07T23:09:21Z" level=info msg="Starting listener" instanceId=i-instanceid listener=spot
time="2020-05-07T23:09:21Z" level=info msg="Starting listener" instanceId=i-instanceid listener=autoscaling
time="2020-05-07T23:09:21Z" level=info msg="Waiting for termination notices" instanceId=i-instanceid
time="2020-05-07T23:09:21Z" level=debug msg="Creating sqs queue" instanceId=i-instanceid listener=autoscaling queue=lifecycled-i-instanceid
time="2020-05-07T23:09:21Z" level=debug msg="Creating sqs queue" instanceId=i-instanceid listener=spot queue=lifecycled-i-instanceid
time="2020-05-07T23:09:21Z" level=debug msg="Subscribing queue to sns topic" instanceId=i-instanceid listener=autoscaling topic="arn:aws:sns:us-east-1:account:terminate-hook-name"
time="2020-05-07T23:09:21Z" level=debug msg="Subscribing queue to sns topic" instanceId=i-instanceid listener=spot topic="arn:aws:sns:us-east-1:account:terminate-hook-name"
time="2020-05-07T23:09:21Z" level=debug msg="Polling sqs for messages" instanceId=i-instanceid listener=spot queueURL="https://sqs.us-east-1.amazonaws.com/account/lifecycled-i-instanceid"
time="2020-05-07T23:09:22Z" level=debug msg="Polling sqs for messages" instanceId=i-instanceid listener=autoscaling queueURL="https://sqs.us-east-1.amazonaws.com/account/lifecycled-i-instanceid"

Sanitized config:

LIFECYCLED_HANDLER=/usr/local/bin/terminate-hook.sh
LIFECYCLED_SNS_TOPIC=arn:aws:sns:us-east-1:account:terminate-hook-name
AWS_REGION=us-east-1

[lox]

Hmmm, that is odd @paulhamby. Did #85 fix the issue for you?

[paulhamby]

Yes, it did. :)