status: 500, error: InternalUncaught, message: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 11746692

[albertnanda]

Hi,

Where is the setting to update the maximum size and the steps to rebuild and update hydrosphere.

Thanks, AN

[albertnanda]

Also, how can I update this setting: "grpc_message":"deadline exceeded after 59999968425ns","grpc_status":4} @KineticCookie @tidylobster

[tidylobster]

Haven't verified this myself, but my guess is that you have to increase the message size parameter at client side. By default it's set to 4MB. Server side has this value set to 1GB (https://github.com/Hydrospheredata/hydro-serving/blob/master/manager/src/main/docker/start.sh#L23).

Try this:

channel = grpc.insecure_channel('localhost', options=[
    ('grpc.max_send_message_length', MAX_MESSAGE_LENGTH), 
    ('grpc.max_receive_message_length', MAX_MESSAGE_LENGTH)
])

[tidylobster]

Not sure about the execution time, though. @KineticCookie can answer more certainly.

[albertnanda]

@tidylobster : Still getting the same error channel = grpc.insecure_channel("localhost:80",options=[('grpc.max_receive_message_length', 2000000000)]) I want to receive a lot of data from server ~20 MB, even after passing additional setting, I get this error: file_line":1036,"grpc_message":"gRPC message exceeds maximum size 4194304: 4293800","grpc_status":8}

[albertnanda]

Setting both the parameters has no effect as well, channel = grpc.insecure_channel("localhost:80",options=[('grpc.max_receive_message_length', 20000000), ('grpc.max_send_message_length', 200000000)])

[albertnanda]

@tidylobster Seems like we need to update the config on server. https://github.com/grpc/grpc/issues/15738 @KineticCookie

[tidylobster]

By the way, what's your setup? Kubernetes or Docker?

[albertnanda]

Docker

[albertnanda]

Just to add a bit more detail, I have two problems viz.

1. Overisized I/O: I want to send something back from the server which is approx ~20 MB
2. I am getting GRPC "grpc_message":"deadline exceeded after 59999968425ns","grpc_status":4} The second problem is a little peculiar, I am calling service B from inside another service A(using GRPC). But if I call service B directly, then I am not getting timed out. The total runtime of serivce B is ~13 mins. @KineticCookie @tidylobster

[tidylobster]

About the second problem - this might be related to this timeout: https://github.com/Hydrospheredata/hydro-serving-gateway/blob/master/src/main/docker/start.sh#L16

Try to restart a gateway service with the bigger GRPC_DEADLINE environment variable.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.