Message length limit

[nabeel-oz]

Description

I am hitting gRPC maximum message size limits when making a SSE call through the Qlik load script.

The following error occurred:
Connector reply error: grpc::StatusCode::RESOURCE_EXHAUSTED: 'Received message larger than max (5106076 vs. 4194304)'

I have already tried increasing the message length on the SSE server using the code below, but this does not seem to help. Is there any way to increase this limit?

MAX_MESSAGE_LENGTH = 50 * 1024 * 1024

server = grpc.server(futures.ThreadPoolExecutor(max_workers=10),\
        options=[('grpc.max_send_message_length', MAX_MESSAGE_LENGTH),('grpc.max_receive_message_length', MAX_MESSAGE_LENGTH)])

Steps to Reproduce

1. Make an SSE call that returns a response > 4 MB.

Expected behavior

Ability to change message length limits.

Actual behavior

Changing message length limit for the SSE does not solve the issue.

Environment

Operating system

[x ] Windows
[ ] Linux

Qlik Sense

[ ] Desktop
[ x] Server

Versions

• server-side-extension: Python SSE derived from Column Operations examples
• Qlik Sense: June 2018
• Operating system: Windows Server 2012 R2
• Python 3.6

[nabeel-oz]

I discovered that the limit was being exceeded due to a bug in my code which was clumping the response into just one bundle.

I was able to get past the limit by splitting the response into an appropriate number of bundles.

# Stream response as BundledRows
for i in range(0, len(response_rows), rows_per_bundle):
     # Yield Row data as Bundled rows
     yield SSE.BundledRows(rows=response_rows[i : i + rows_per_bundle])

[qlik-aoh]

Hi @nabeel-oz,

Great that you found a solution to your problem!

It could actually occur also with this solution if the rows in your response is much larger than the rows in your request. In our examples we assume this is not the case and responds with one Bundledrows per requested BundledRows. A production environment could benefit adding some logic to possibly split the response on several bundles.

Best regards, Markus

[nabeel-oz]

Hi @qlik-aoh

Thanks I'll keep that in mind. Do you know how Qlik calculates the number of bundles for the request? I suppose the logic would need to consider the memory footprint per row in the response.

Best regards, Nabeel

[qlik-aoh]

Hi again @nabeel-oz,

Qlik will add rows to the bundle until the expected message size is around 64 KiB. This has shown the best throughput in our tests and follows the recommendation from the gRPC developers.

Best regards, Markus

[vc2014]

Hi qlik-aoh,

I am facing a similar issue while sending the response back from QLIK SSE to QLIK Sense. I am using QLIK SSE java component. Could you kindly help by providing a solution similar to python yeild method.

Thanks, Vishal Chauhan