request failed, status: 500 Internal Server Error

[N1k-3l]

Describe the bug When try to get metrics from group of hosts (more than 50) --> Error 'request failed, status: 500 Internal Server Error' Can it be from plugin error, because with old (3version) plugin it worked normal?

Log

t=2021-12-04T16:09:53+0300 lvl=dbug msg="Publication successful" logger=live user=16 channel=grafana/dashboard/uid/dtx8-Ednk
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Querying for data source via SQL store" logger=datasources id=5 orgId=1
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Initializing new data source instance" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Zabbix request" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource method=apiinfo.version
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Got Zabbix version" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource version=54.000
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Zabbix request" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource method=apiinfo.version
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Zabbix request" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource method=hostgroup.get
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Successfully authenticated" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource url=http://ZABBBIX_XXX/api_jsonrpc.php user=XXX
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Zabbix request" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource method=hostgroup.get
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Writing result to cache" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource method=hostgroup.get
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Zabbix request" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource method=host.get
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Writing result to cache" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource method=host.get
t=2021-12-04T16:09:53+0300 lvl=dbug msg="Zabbix request" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource method=item.get
t=2021-12-04T16:09:59+0300 lvl=eror msg="Zabbix API request error" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource error="request failed, status: 500 Internal Server Error"
t=2021-12-04T16:09:59+0300 lvl=eror msg="Request Completed" logger=context userId=16 orgId=1 uname=XXXXX method=POST path=/api/datasources/5/resources/zabbix-api status=500 remote_addr=XXX.XXX.XXX.XXX time_ms=5375 size=95 referer="https://grafana.XXX.ru/d/dtx8-Ednk/network-interfaces-utilization?orgId=1&editPanel=4"

Screenshots

request failed, status: 500 Internal Server Error Object status:500 statusText:"Internal Server Error" data:Object error:"Internal Server Error" message:"request failed, status: 500 Internal Server Error" config:Object url:"api/datasources/5/resources/zabbix-api" method:"POST" headers:Object hideFromInspector:false data:Object retry:0 message:"request failed, status: 500 Internal Server Error"

Software versions

Grafana	Zabbix	Grafana-Zabbix Plugin
8.2.5	5.4.8	4.2.4

[mbuyukkarakas]

Hello. I have the same problem.

[alexanderzobnin]

It looks like Zabbix API cannot handle request. Try to tweak PHP settings and increase amount of memory assigned to PHP. I appreciate if you can make the same request in grafana with 3.x plugin version to make sure it's something that works in a different way in the new plugin versions with backend.

[brezzii]

Hello @alexanderzobnin why it was working before with the same php settings as am using now ? does the amount of data affects the API Call ? Regards

[brezzii]

By the way our php memory settings are 10 G and we still have the same issue

[N1k-3l]

It looks like Zabbix API cannot handle request. Try to tweak PHP settings and increase amount of memory assigned to PHP. I appreciate if you can make the same request in grafana with 3.x plugin version to make sure it's something that works in a different way in the new plugin versions with backend.

I tweaked up PHP settings, but grafana doesn't get metrics, only freeze and stop working during 2-5 minutes. No errors, but no metrics.

[jmrodriguezqtn]

I have the same problem, still no solution?

[alexanderzobnin]

Hm, that's strange, I will try to investigate. Can you provide a bit more info, is it happening only on large amount of data?

[hotoroy]

Hi! I have same error. I'm trying to get data on 1 item for 150 hosts.

[N1k-3l]

Hm, that's strange, I will try to investigate. Can you provide a bit more info, is it happening only on large amount of data?

Yes. As I wrote in the description, this problem only occurs when the group contains over ~50 hosts.

[GOID1989]

Have same problem on zabbix 6.0.3 + grafana 8.5.2 + plugin 4.2.8 Does not matter with OR without "direct db"

But! Identical panel (table type) with DS connected to zabbix 5.0.10 work correct

After changing php param php_value[memory_limit] = 1024M few panel showing correct, but not all.

Php error log when choosing memory level: `[15-Jun-2022 21:14:25] WARNING: [pool zabbix] child 3288729 said into stderr: "NOTICE: PHP message: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /usr/share/zabbix/include/classes/api/CRelationMap.php on line 48"

[15-Jun-2022 21:15:10] WARNING: [pool zabbix] child 3289012 said into stderr: "NOTICE: PHP message: PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /usr/share/zabbix/include/db.inc.php on line 404"`

Maybe problem in filter passed by grafana plugin ? Because on this zabbix i have bunch of scripts grabs info by API - all host, all host trigger, all host items at same time and it works.

[huornlmj]

I have the same issue when trying to poll a large amount of hosts with a single item per host. If I select a time window of 3 hours I get my data, if I extend that to 6 hours I get error 500 and "no data" (error:"request failed, status: 500 Internal Server Error"). I don't know if it's Grafana or Zabbix that's throwing this 500 error as it's not showing up in either of their log files.

[huornlmj]

UPDATE - I found that if I decreased the frequency at which Zabbix polls the monitored hosts from every 1 minute to every 5 minutes and leave it for 24 hours, I can now go back to Grafana and graph those hosts up as far as 12 hours. But anything more than 12 hours (e.g. 24 hours) and I start getting the error 500 again. It must be to do with the volume of data being retrieved.

[rfrozza]

I was receiving the same error, but increasing the parameter php_value memory_limit to 1024M inside zabbix.conf file resolved the issue for me. /etc/apache2/conf-enabled/zabbix.conf Tested with a dashboard with +200 hosts and 6 hours as time range Zabbix v6.2.3 and Grafana v9.2.5 And the setup is running on AWS, with RDS for MySQL and two EC2 instances (one for zabbix server and one for zabbix frontend).

[morelattes]

@alexanderzobnin Still facing this issue on grafana-zabbix 4.2.10, grafana 8.0 and zabbix 6.2 (tried everything mentioned in the comments). Any commits related to this?

[Infant-Rashmi]

@alexanderzobnin facing the same issue, on grafana-zabbix 4.2.10, grafana 9.4.7 and zabbix 6.2. Could you please look into it?

[rmsairlock]

@alexanderzobnin same issue, grafana-zabbix 4.4.1, grafana 10.1, zabbix 6.4.6

[rmsairlock]

Just wondering, was this ever figured out? I still have the issue when getting a value from around 400 servers, if the host isn't a wild card the query/results run fine.

[Rudorufu-san]

same issue, grafana-zabbix 4.4.9, grafana 11.0, zabbix 6.4.15 Edit sudo nano /etc/postgresql/16/main/postgresql.conf

• Increase from 11MB to work_mem = 64MB Editing PHP.INI /etc/php/7.4/apache2/php.ini and /etc/php/7.4/cli/php.ini
• memory_limit = 2G Modification in apache config /etc/apache2/apache2.conf

  Options FollowSymLinks AllowOverride None Require all denied

  <Directory /usr/share>

  AllowOverride None

  Require all granted

  <Directory /usr/share> AllowOverride All Require all granted <Directory /usr/share/zabbix> AllowOverride All Require all granted <Directory /var/www/> Options Indexes FollowSymLinks AllowOverride None Require all granted

  <Directory /srv/>

  Options Indexes FollowSymLinks

  AllowOverride None

  Require all granted

  sudo nano /usr/share/zabbix/.htaccess created and supplemented by

• php_value memory_limit 2G Searching the root of Apache because it didn't work for me by exploring the www /etc/apache2/sites-available/000-default.conf File info.php and test_memory.php created in /usr/share/zabbix folder. To check the settings changes when debugging a fault Checking the apache error log and adjusting the memory requirements in two php scripts by adding values /usr/share/zabbix/include/ -Db.inc.php -CJsonRpc.php
• ini_set('memory_limit', '2G'); Still in /etc/apache2/conf-enable/zabbix.conf php_value max_execution_time 30000 php_value memory_limit 4G php_value post_max_size 160M php_value upload_max_filesize 100M php_value max_input_time 30000 php_value max_input_vars 12000000 php_value always_populate_raw_post_data -1 php_value max_execution_time 30000 php_value memory_limit 4G php_value post_max_size 160M php_value upload_max_filesize 100M php_value max_input_time 30000 php_value max_input_vars 12000000 php_value always_populate_raw_post_data -1

  Unfortunately, the error still persists. I think tha is limitation of Zabbix API it borders.

On item i set preprocesing throtling with hard beat. wait what its do in a month. I like build graph with AVG of all status port on switch with 70p metric per 1m from last 30d and 90d :-)