No notification due to webhook timeout

[hitsub2]

Describe the bug With some webhook, like slack or feishu, there is 100% failure for feishu and some failure for slack, due to the timeouts. Same with issue #534 .

Steps to reproduce Using FIS to simulate the spot int.

Expected outcome A notification that a spot instance will be terminated.

Application Logs

ERR Webhook Error: Client Do failed error="Post \"[https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxxxxxxxxxx\](https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxx/) (https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxx/)": 
context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Environment

• NTH App Version: 1.17.3
• NTH Mode (IMDS/Queue processor): IMDS
• OS/Arch: x86
• Kubernetes version: 1.22
• Installation method: helm

NTH Configration

webhookHeaders: '{"Content-type":"application/json"}'
webhookProxy: ""
webhookTemplate: '{"msg_type": "text","content": {"text": "EventID: {{ .EventID }} - Kind: {{ .Kind }} - Instance: {{ .InstanceID }} - Node: {{ .NodeName }} - Description: {{ .Description }} - Start Time: {{.StartTime }}"}}'
webhookTemplateConfigMapKey: ""
webhookTemplateConfigMapName: ""
webhookURL: "https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxx"

[snay2]

Thanks for the issue! I've re-opened the original issue #534 and marked it not stale; let's continue the discussion there.