SNOW-1437885 Disable blob interleaving when in Iceberg mode

snowflakedb / snowflake-ingest-java

Java SDK for the Snowflake Ingest Service -

http://www.snowflake.net

Apache License 2.0

69 stars 54 forks source link

SNOW-1437885 Disable blob interleaving when in Iceberg mode #763

Closed sfc-gh-alhuang closed 3 months ago

sfc-gh-alhuang commented 4 months ago

Added a method to SnowflakeStreamingIngestClientInternal to activate/deactivate streaming to Iceberg tables.
A blob should only contain one chunk (parquet) under Iceberg mode. Disable blob interleaving under Iceberg mode.

sfc-gh-alhuang commented 4 months ago

Thanks for the comments and tips! Added icebergMode on/off params for some unit tests. Some tests included blob registration and authentication only test wit regular mode for now. @sfc-gh-tzhang I plan to delete or modify the client configure call in later PRs as it's only called by StreamingIngestStage and we might not use this class in the iceberg mode later. The Iceberg client & table compatibility check is adding in the server side (ref), should we also add a check in the client?

sfc-gh-alhuang commented 3 months ago

Per offline discussion with @sfc-gh-tzhang , moved MAX_CHUNKS_IN_BLOB_AND_REGISTRATION_REQUEST_ICEBERG_MODE_DEFAULT back to ParameterProvider.java.