aws / aws-node-termination-handler

Gracefully handle EC2 instance shutdown within Kubernetes
https://aws.amazon.com/ec2
Apache License 2.0
1.63k stars 266 forks source link

No notification due to webhook timeout #724

Closed hitsub2 closed 1 year ago

hitsub2 commented 1 year ago

Describe the bug With some webhook, like slack or feishu, there is 100% failure for feishu and some failure for slack, due to the timeouts. Same with issue #534 .

Steps to reproduce Using FIS to simulate the spot int.

Expected outcome A notification that a spot instance will be terminated.

Application Logs

ERR Webhook Error: Client Do failed error="Post \"[https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxxxxxxxxxx\](https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxx/) (https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxx/)": 
context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Environment

NTH Configration

webhookHeaders: '{"Content-type":"application/json"}'
webhookProxy: ""
webhookTemplate: '{"msg_type": "text","content": {"text": "EventID: {{ .EventID }} - Kind: {{ .Kind }} - Instance: {{ .InstanceID }} - Node: {{ .NodeName }} - Description: {{ .Description }} - Start Time: {{.StartTime }}"}}'
webhookTemplateConfigMapKey: ""
webhookTemplateConfigMapName: ""
webhookURL: "https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxx"
snay2 commented 1 year ago

Thanks for the issue! I've re-opened the original issue #534 and marked it not stale; let's continue the discussion there.