Zulip Zabbix integration : setup the weebhook to send data in the JSON format

[ali-yahyaoui]

Hi , i Just set a incoming webhook from zabbix for web monitoring , but Zulip bot send me this message " Hi there! It looks like you tried to setup the Zulip Zabbix integration, but didn't correctly configure the webhook to send data in the JSON format that this integration expects!" how i can setup the weebhook to send data in the JSON format ?

[timabbott]

Can you post a screenshot of the Zabbix UI you're looking at? I think Zabbix may have a menu for what format to post in, but I don't recall.

[zulipbot]

Hello @zulip/server-api members, this issue was labeled with the "area: documentation (api and integrations)" label, so you may want to check it out!

[timabbott]

@Hypro999 FYI.

[Hypro999]

@ali-yahyaoui did you follow the steps listed at https://chat.zulip.org/integrations/doc/zabbix to set up your integration? For the Zabbix integration, the payload needs to follow the following formats:

subject:

{TRIGGER.STATUS}-{TRIGGER.SEVERITY}-{TRIGGER.NAME}

message:

{
    "hostname": "{HOST.NAME}",
    "severity": "{TRIGGER.SEVERITY}",
    "status": "{TRIGGER.STATUS}",
    "item": "{ITEM.NAME1} is {ITEM.VALUE1}",
    "trigger": "{TRIGGER.NAME}",
    "link": "https://zabbix.example.com/tr_events.php?triggerid={TRIGGER.ID}&eventid={EVENT.ID}"
}

If you still get an error message after following the listed steps and enforcing these payload formats, let me know and I'd be happy update the integration.

[ali-yahyaoui]

@timabbott @Hypro999 yes, I followed all the steps listed !

[Hypro999]

This seemed pretty weird so I actually took the time to setup a sort of "Zabbix lab" today (I'm new to Zabbix so this took longer than expected) and I wasn't able to replicate this issue. I tested the webhook with the Template OS Linux by Zabbix agent/High CPU utilization (over {$CPU.UTIL.CRIT}% for 5m) trigger and the payload I got was perfectly valid JSON

{
  "hostname": "Monitored Server",
  "severity": "Warning",
  "status": "PROBLEM",
  "item": "CPU utilization is 100 %",
  "trigger": "High CPU utilization (over 90% for 5m)",
  "link": "https://zabbix.hypro999.engineer/tr_events.php?triggerid=16682&eventid=90"
}

But interestingly enough the content type header application/x-www-form-urlencoded. So to see if this would be an issue, I tried it again and this time on a live Zulip server instance (chat.zulip.org) and the notification came through just fine. So I'm pretty sure that the Zabbix webhook code on Zulip's end is not the cause of this error.

If you used the test feature and sent non-JSON data or JSON data with missing keys then you could accidently trigger this message (like I once did when testing). So @ali-yahyaoui could you provide me with more information like what trigger or action from Zabbix's end might have caused this message?

The message:

What I meant by the test feature:

[ali-yahyaoui]

@Hypro999 I am very grateful, thank you for your time and effort. Yes me too i use the same template Template OS Linux by Zabbix agent/High CPU utilization (over {$CPU.UTIL.CRIT}% for 5m) and i receive all the notifications correctely also about other trigger like Load average trigger and Nginx trigger ...

But the problem is about notification from trigger Web Test Fail for the web monitoring , I set this trigger and Web Scenario to check URL endpoint and response code on every host, please take a look here https://www.zabbix.com/documentation/4.4/manual/web_monitoring, the trigger and web scenario work correctely when i test to get notification, i get this message on Zulip

[Hypro999]

@ali-yahyaoui I see, thanks for the detailed explanation of what's going on on your end. Could you do me a favor? I need to see what payload is being sent to the server but setting up the same exact environment you're working with would take too much time for me. So I need you to help me determine what that payload is. Here's how: if you temporarily change your zulip.sh script from

#!/bin/bash
webhook_url="$1"
payload="$3"
/usr/bin/curl -m 5 --data "$payload" "${webhook_url}"

to

#!/bin/bash
webhook_url="$1"
payload="$3"
echo $payload > /tmp/payload.txt  # this is the new line I want you to add temporarily
/usr/bin/curl -m 5 --data "$payload" "${webhook_url}"

Then you should be able to capture the payload data to /tmp/payload.txt. After you do that for the event that's causing this issue, if you could post that content here (you should redact or fake any sensitive/personal information you find first in the payload) it would really help!

[Hypro999]

@ali-yahyaoui any updates or should we close this issue?

[GhaziTriki]

@Hypro999 I am working with @ali-yahyaoui, I will update our zulip to gather those logs. Please wait before closing it.

[GhaziTriki]

@ali-yahyaoui I updated "alertscripts/zulip.sh", could you please do more testing and then we will be able to share the output.

[GhaziTriki]

@Hypro999

Here is a sample output

{
 "hostname": "BBB meeting 01",
 "severity": "Disaster",
 "status": "PROBLEM",
 "item": "Failed step of scenario "Status". is 1",
 "trigger": "Web Test Fail",
 "link": "https://********.***/tr_events.php?triggerid=27293&eventid=250037"
 }

[Hypro999]

Ah, I see the issue here. This is indeed invalid JSON.

"item": "Failed step of scenario "Status". is 1",

Here Zabbix is not escaping the " marks around "Status" and this is causing the string to be terminated early with the rest of the string being syntactically invalid junk. You can verify that this is indeed invalid JSON according to RFC 8259 via. online tools like https://jsonformatter.curiousconcept.com/.

At this point there's nothing we can do from Zulip's end. This is an issue either with Zabbix or your configuration and you'll have to try to get this fixed upstream.

[Hypro999]

And just a thought, if the value for item is something that can be manipulated by a user or external entity, you may be enabling a potential injection attack here, because values for the same key in JSON are overridden by the last instance. e.g.

{
 "hostname": "BBB meeting 01",
 "severity": "Disaster",
 "status": "PROBLEM:", "status": "overriden"
}

Would be loaded as

{'hostname': 'BBB meeting 01', 'severity': 'Disaster', 'status': 'overriden'}

[GhaziTriki]

@Hypro999 We are using zabbix 4.4.4, what version are you using?

[Hypro999]

@GhaziTriki I tested this code against the same version.

[Hypro999]

To be more specific I used https://marketplace.digitalocean.com/apps/zabbix to test. So the version of Zabbix is kind of irrelevant here.

[timabbott]

@GhaziTriki you may be able to see the exact JSON your server is sending in /var/log/zulip/webhook_errors.log. Can you try checking that?

[GhaziTriki]

@timabbott Now we have upgraded to Zabbix 5.0 LTS and mastering more of its features. The error have happens at a "DISASTER" as example when we hare deboot the server.

The problem happens because of an unscaped " character.

The bad JSON generated by Zabbix

{
"hostname": "LBs",
"severity": "Disaster",
"status": "PROBLEM",
"item": "Last error message of scenario "Status". is Couldn't connect to server: Failed to connect to vclb2 port 443: Connection refused",
"trigger": "Web scenario "Status" failed: Couldn't connect to server: Failed to connect to vclb2 port 443: Connection refused",
"link": "https://zabbix.domain.net/tr_events.php?triggerid=33901&eventid=7540811"
}

The good JSON should be

{
"hostname": "LBs",
"severity": "Disaster",
"status": "PROBLEM",
"item": "Last error message of scenario \"Status\". is Couldn't connect to server: Failed to connect to vclb2 port 443: Connection refused",
"trigger": "Web scenario \"Status\" failed: Couldn't connect to server: Failed to connect to vclb2 port 443: Connection refused",
"link": "https://zabbix.domain.net/tr_events.php?triggerid=33901&eventid=7540811"
}

Is a Zabbix issue? Can we fix the JSON formatting in the zabbix shell script integration?

[Phfas]

Heya, Just ran into this problem and I think i've worked around this issue like this:

Update the json string like this:

{
    'hostname': '{HOST.NAME}',
    'severity': '{TRIGGER.SEVERITY}',
    'status': '{TRIGGER.STATUS}',
    'item': '{ITEM.NAME1} is {ITEM.VALUE1}',
    'trigger': '{TRIGGER.NAME}',
    'link': 'https://zabbix.example.com/tr_events.php?triggerid={TRIGGER.ID}&eventid={EVENT.ID}'
}

and then use some bash string replacement in the zulip.sh we'll replace the json single quotes in the bash script after we replace the double quotes in the values.

#!/bin/bash
webhook_url="$1"

# Build our JSON payload and send it as a POST request to the Zulip incoming web-hook URL
# replace double quotes with escaped double quotes
payload=${3//\"/\\\"}
# replace single quotes with double quotes
newpayload=${payload//\'/\"} 
/usr/bin/curl -m 5 --data "$newpayload" "${webhook_url}"

[GhaziTriki]

@Phfas Thank you, we will test it and give a feedback.

[adambirds]

@GhaziTriki if you update to zabbix 5.2. And follow my changes to the Zabbix Docs that where merged into zulip/master in #17632 yesterday. This should solve your issue. We move away from the bash script to send the payload and use zabbix's new built in webhook feature that uses javascript to send the payload. This should resolve. Let me know if it doesn't. As I put the javascript on the zabbix side and the zabbix integration for zulip together in the first place so I should be able to test as I have a zabbix server working with zulip.

You don't need to update zulip to master to try this, you just need to follow the new docs as the only changes are to the zabbix side, we completely move away from the bash script.