failed to receive call resource response: rpc error: code = ResourceExhausted desc = grpc: received message larger than max

[cronop]

Intermittent error when running a topN time series query on item utilization across a large number (>1000) of switch interfaces.

Similar to https://github.com/alexanderzobnin/grafana-zabbix/issues/1087

grafana-zabbix 4.2.4 grafana Version 8.1.6 (commit: 4a4083716c, branch: HEAD) mysqld Ver 10.5.12-MariaDB for Linux on x86_64 (MariaDB Server) zabbix_server (Zabbix) 5.0.15 aws linux 4.14.246-187.474.amzn2.x86_64

grafana.log lvl=eror msg="Failed to call resource" logger=context userId=1 orgId=1 uname=admin error="failed to receive call resource response: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (17089470 vs. 16777216)" lvl=eror msg="Failed to call resource" logger=context userId=4 orgId=1 uname=admin error="failed to receive call resource response: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (21246971 vs. 16777216)"

JSON response Failed to call resource Object status:500 statusText:"Internal Server Error" data:Object message:"Failed to call resource" config:Object url:"api/datasources/2/resources/db-connection-post" method:"POST" headers:Object hideFromInspector:false data:Object retry:0 message:"Failed to call resource"

[iRomanyshyn]

I've got the same error when trying to do a big requests. It looks like some settings can be changed on the remote database side to extend possible size of the message.

[cronop]

I've got the same error when trying to do a big requests. It looks like some settings can be changed on the remote database side to extend possible size of the message.

добрый ранок, any ideas which settings?

[iRomanyshyn]

I've got the same error when trying to do a big requests. It looks like some settings can be changed on the remote database side to extend possible size of the message.

добрый ранок, any ideas which settings? We have no "ы" letter in Ukrainian alphabet, BTW :)

I'm working on it. But maybe we're just asking too much from it. I got that problem while trying to process a thousands of metrics into a single gauge with one month length. Which is pretty similar to your case. Also, which DB are you using for Zabbix? And did you made a direct connection for it?

[cronop]

so sorry about the copypasta! i am not sure whether i know less golang or ukrainian I *think the issue is that the request code has set the grpc message size to 16MB (based on the errors) and it needs to be increased for our use cases ? but could be wrong https://stackoverflow.com/questions/54137846/updating-receive-and-send-message-size-for-grpc-in-golang

[banzayats]

I have the same problem: when I try to show data for a large period, I get an error "Failed to call resource":

Object
status:500
statusText:"Internal Server Error"
data:Object
message:"Failed to call resource"
config:Object
url:"api/datasources/2/resources/db-connection-post"
method:"POST"
headers:Object
hideFromInspector:false
data:Object
retry:0
message:"Failed to call resource"

In the Grafana log i see this:

t=2022-01-20T13:27:30+0000 lvl=eror msg="Failed to call resource" logger=context userId=2 orgId=1 uname=vk error="failed to receive call resource response: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (27700670 vs. 16777216)"

grafana-zabbix 4.2.4 grafana v8.3.1 (8d38082755) zabbix 5.4.4

Is this a limitation of the Grafana itself (but then it is not clear why much more data is loaded from the elastic data source) or the plugin?

[AlexFavre]

Hi everybody,

We have the same issue but after contact with grafana support they have say that's this issue is plugin side also how we can fix this parameter (grpc max received message size)? Zabbix 6.0.7 grafana 9.0.2 plugins 4.2.9

[Ricaz]

Two years later and I'm running into the same issue. This must be related to the Go part of the plugin, which uses gRPC. Would be nice to be able to configure this!

To clarify, I'm doing a query for all interfaces on ~500 hosts, which sometimes exceeds the 16MiB limit.

[zoltanbedi]

This is currently not configurable but we will check if we want to expose this option to the users.

[Ricaz]

This is currently not configurable but we will check if we want to expose this option to the users.

That would be nice. I see no performance impact on Zabbix or Postgres when making these large queries, so it's only the gRPC queries (and Grafana) holding back.