Recieving error: `message length too large`

[maxsargentdev]

Hi,

When querying certain quickwit indexes using the grafana plugin I sometimes receive messages in the UI like so:

internal error: `Error, message length too large: found 39354854 bytes, the limit is: 20971520 bytes`

I have enabled debug logs in grafana but all I can see is:

logger=plugin.quickwit-quickwit-datasource t=2024-06-22T20:39:04.633969392Z level=debug msg=CallResource url="http://quickwit-searcher.quickwit-system:7280/api/v1/_elastic/<index id>/_field_caps?start_timestamp=1718483944&end_timestamp=1719088745"

But no errors in grafana itself. I am guessing because this error is being bubbled up via the plugin.

Is it possible that we could make both the returned bytes and the display limit configurable and defaulting to the same value?

I can see in the latest grafana sdk packages that there is some middleware for limiting the response back from datasources:

https://pkg.go.dev/github.com/grafana/grafana-plugin-sdk-go/backend/httpclient#ResponseLimitMiddleware

I am willing to make the change if one is required I just need some direction.

Thanks, Max Sargent

[guilload]

Hello @maxsargentdev,

This is an issue happening at the transport level when nodes communicate together. Some gRPC messages are too large. When encountering the problem, is the last debug log always /api/v1/_elastic/<index id>/_field_caps? Does it go away when reducing the time span of the query to a few hours? Do the problematic indexes have more complex schemas?

It should go away by increasing the gRPC max message size using the config property grpc.max_message_size: https://quickwit.io/docs/configuration/node-config#grpc-configuration

[maxsargentdev]

All of them are true.

Apologies I should have found this in the documentation, I will try it out today.

Thanks for all the work on quickwit it's a great project!

[guilload]

Ok, keep me posted.

[maxsargentdev]

this has worked a treat, thanks again!