Managed streaming - better estimation for decision for streaming vs queuing

ohadbitt commented 4 months ago

A new heuristic for choosing between queuing and streaming in Managed streaming client, the default behavior is using the an estimation against the 4mb limit after dividing or multiplying by some factor described by the consts:

// Regardless of the format, we don't want to stream more than 10mb int MAX_STREAMING_STREAM_SIZE_BYTES = 10 1024 1024; // Used against the users input of raw data size int MAX_STREAMING_RAW_SIZE_BYTES = 6 1024 1024; double JSON_UNCOMPRESSED_FACTOR = 1.5d; int NON_BINARY_FACTOR = 2; double BINARY_COMPRESSED_FACTOR = 2d; double BINARY_UNCOMPRESSED_FACTOR = 1.5d;

This will also allow users to stream bigger than 4mb non-compressed data

disableRetries option to client options - default is true only for streaming clients as if stream is not repeatable it will simply fails - better to let the user deal with it (alternative considered was to always create repeatable streams - like in managed streaming)
better error messages

github-actions[bot] commented 4 months ago

Test Results

326 tests +4 318 ✅ +4 3m 50s ⏱️ +48s 27 suites +2 8 💤 ±0 27 files +2 0 ❌ ±0

Results for commit 304ff39e. ± Comparison against base commit b7d2477f.

:recycle: This comment has been updated with latest results.

AsafMah commented 4 months ago

This is a very big PR, can you summarize the changes in more detail?

Also, for disableRetries, maybe we want to fold this in to a more robust Policy object like @yogilad suggested and worked on.

AsafMah commented 3 weeks ago

Go over the pending comments, but approved in principle