search

snowflakedb / snowflake-ingest-java

Java SDK for the Snowflake Ingest Service -
http://www.snowflake.net
Apache License 2.0
65 stars 51 forks source link

SNOW-1512047 Introduce independent per-table flushes when interleaving is disabled #788

Open sfc-gh-alhuang opened 3 weeks ago

sfc-gh-alhuang commented 3 weeks ago

The current SDK behavior flushes all channels simultaneously when any buffer reaches its limit, potentially causing unnecessary small file flushes if interleaving is disabled (MAX_CHUNKS_IN_BLOB_AND_REGISTRATION_REQUEST = 1) and the ingestion throughput between tables is uneven.

As the WIP streaming to Iceberg table feature set MAX_CHUNKS_IN_BLOB_AND_REGISTRATION_REQUEST = 1. This PR introduces per-table flushing to avoid the above issue.

JIRA