search

ydb-platform / ydb-python-sdk

YDB Python SDK
https://ydb-platform.github.io/ydb-python-sdk/
Apache License 2.0
87 stars 50 forks source link

bug: CLIENT: Sent message larger than max (64975458 vs. 64000000) with bulk upsert #460

Open maximyurchuk opened 4 months ago

maximyurchuk commented 4 months ago

Bug Report

YDB Python SDK version:

maxim-yurchuk@yurchuk-github:~/ydb/ydb$ pip show ydb Name: ydb Version: 3.11.3

Environment

maxim-yurchuk@yurchuk-github:~/ydb/ydb$ uname -a
Linux yurchuk-github 5.15.0-107-generic https://github.com/ydb-platform/ydb-python-sdk/issues/117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Current behavior:

Huge bulk upsert (driver.table_client.bulk_upsert) gives the following error 

2024-07-12T08:09:49.8675916Z Traceback (most recent call last):
2024-07-12T08:09:49.8677299Z   File "/home/runner/.local/lib/python3.10/site-packages/ydb/connection.py", line 458, in __call__
2024-07-12T08:09:49.8678065Z     response = rpc_state(
2024-07-12T08:09:49.8678879Z   File "/home/runner/.local/lib/python3.10/site-packages/ydb/connection.py", line 242, in __call__
2024-07-12T08:09:49.8679773Z     response, rendezvous = self.rpc.with_call(*args, **kwargs)
2024-07-12T08:09:49.8680722Z   File "/home/runner/.local/lib/python3.10/site-packages/grpc/_channel.py", line 1198, in with_call
2024-07-12T08:09:49.8681604Z     return _end_unary_response_blocking(state, call, True, None)
2024-07-12T08:09:49.8682681Z   File "/home/runner/.local/lib/python3.10/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
2024-07-12T08:09:49.8683773Z     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
2024-07-12T08:09:49.8684656Z grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
2024-07-12T08:09:49.8685362Z    status = StatusCode.RESOURCE_EXHAUSTED
2024-07-12T08:09:49.8686015Z    details = "CLIENT: Sent message larger than max (64975458 vs. 64000000)"
2024-07-12T08:09:49.8687739Z    debug_error_string = "UNKNOWN:Error received from peer  ***grpc_message:"CLIENT: Sent message larger than max (64975458 vs. 64000000)", grpc_status:8, created_time:"2024-07-12T08:09:49.841712345+00:00"***"
2024-07-12T08:09:49.8688817Z >
        """

Expected behavior:

No any error

Steps to reproduce: Do huge bulk upsert

vgvoleg commented 4 months ago

64*10^6 is the default value for grpc channel params: grpc.max_send_message_length grpc.max_receive_message_length

It is possible to override them: DriverConfig has a special field channel_options. Place a dict with these keys and values you need (-1 could be used for no limits).