How can I get the termination information via webhook?

[HankChow]

I have deployed aws-node-termination-handler as DaemonSet in an EKS cluster. I set the configuration of flags in order to receive notification via webhook when a spot instance terminates. To test whether the flags will work, I only set the flag WEBHOOK_URL with a url which is listening for HTTP requests and leave blanks for other flags as default. I did receive an POST request from a terminating spot instance in EKS cluster terminates. It is expected a JSON like {"text":"[NTH][Instance Interruption] EventID: {{ .EventID }} - Kind: {{ .Kind }} - Description: {{ .Description }} - State: {{ .State }} - Start Time: {{ .StartTime }}"} which provided by the official documentation. But I can't see anything in the request payload.

[bwagner5]

The default webhook request template is for slack webhooks. Depending on what server you're sending the request to, you may need to change the template to match the expected request body.

There's an example in our end-to-end test suite where we change the template to the expected request structure for amazon chime webhooks: https://github.com/aws/aws-node-termination-handler/blob/b6477836cc81f6c2e82ca9840adf170472bbd0fc/test/e2e/webhook-test#L30

[ulisse31]

I don't know if it makes sense to post it in this thread, but our webhook is set to send notifications to Slack, and some of the fields are empty. Is it supposed to be this way?

[bwagner5]

@ulisse31 I believe you're seeing the same error as this issue https://github.com/aws/aws-node-termination-handler/issues/104. This was fixed in v1.3.0, so if you're on a version pre v1.3.0, upgrading to v1.3.1 or to the latest v1.4.0 release should fix it for you.

[HankChow]

@bwagner5 Thanks for your answering. The request is sent to an application based on Flask. I can see that the request does reach my Flask application with POST method, but there is nothing in the payload of the request.

[ulisse31]

@ulisse31 I believe you're seeing the same error as this issue #104. This was fixed in v1.3.0, so if you're on a version pre v1.3.0, upgrading to v1.3.1 or to the latest v1.4.0 release should fix it for you.

@bwagner5 We're already running v1.3.1 on EKS 1.14. We have planned to upgrade 1.4.0 so I'll give it a try and in case report back. Thank you for your reply!

[bwagner5]

@HankChow I did a test with a flask server:

Flask Source Code

$ cat flaskapp.py
from flask import Flask
from flask import request
app = Flask(__name__)

@app.route('/', methods=['POST'])
def index():
    print(request.data)
    return

aws-node-termination-handler local binary

$ git clone git@github.com:aws/aws-node-termination-handler.git
$ cd aws-node-termination-handler
$ go build cmd/node-termination-handler.go
$ ./node-termination-handler --dry-run --node-name test --webhook-url http://localhost:5000 --metadata-url http://localhost:1338

EC2 Metadata Mock

$ curl -Lo ec2-metadata-mock https://github.com/aws/amazon-ec2-metadata-mock/releases/download/v0.9.0/amazon-ec2-metadata-mock-$(uname | tr '[:upper:]' '[:lower:]')-amd64
$ chmod +x ec2-metadata-mock
$ ./ec2-metadata-mock spotitn
...
2020/05/14 14:38:02 Initiating amazon-ec2-metadata-mock for EC2 Spot interruption notice on port 1338
...

Flask Stdout Log

$ pip3 install flask
$ export FLASK_APP="flaskapp.py"
$ flask run
 * Serving Flask app "flaskapp.py"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
b'{"text":"[NTH][Instance Interruption] EventID: spot-itn-1229fe2a46345395a6df93665219d2f184213a5edefed2092a2343c2ff797b63 - Kind: SPOT_ITN - Description: Spot ITN received. Instance will be interrupted at 2020-05-14T19:34:54Z \n - Start Time: 2020-05-14 19:34:54 +0000 UTC"}'

Printing request.data from the flask handler shows the webhook data.

[HankChow]

@bwagner5 Thanks for your answering. I get the webhook data from request.data successfully. It was my mistake that I tried to get the webhook data from request.json.