ydb-platform / ydb-python-sdk

YDB Python SDK
https://ydb-platform.github.io/ydb-python-sdk/
Apache License 2.0
87 stars 50 forks source link

Support data limit when reading a batch with TopicReaderSync #431

Open JasonRammoray opened 4 months ago

JasonRammoray commented 4 months ago

Allow a client to control the amount of data it receives, when reading a batch through TopicReaderSync.

Pull request type

Please check the type of change your PR introduces:

What is the current behavior?

TopicReaderSync.receive_batch ignores max_messages and max_bytes parameters, which means a client has no control over the amount of received data.

Issue Number: 365

What is the new behavior?

TopicReaderSync.receive_batch now takes max_messages and max_bytes into account.

Other information

Decisions made:

  1. Enforce at least one message and at least a single byte on a batch.
  2. Since a batch could theoretically be empty, don't use _commit_get_partition_session, and rather copy _partition_session from the batch to a new (sliced) batch.
  3. Make no slice in case if neither max_messages, nor max_bytes were provided.