search

snowflakedb / snowflake-ingest-java

Java SDK for the Snowflake Ingest Service -
http://www.snowflake.net
Apache License 2.0
71 stars 57 forks source link

SNOW-1754295 - Start using subscoped tokens for iceberg ingestion #868

Closed sfc-gh-hmadan closed 2 weeks ago

sfc-gh-hmadan commented 3 weeks ago
  1. Add logic for invoking new server-side API (Refresh Table Information)
  2. Reuse InternalStage for all the JDBC interop. For this:
    1. pass in a tableRef into InternalStage
    2. Add a IStorageManager.getRefreshedLocation API that's called by InternalStage
  3. Add SubscopedTokenExternalVolumeManager that creates an InternalStage per table, and fetches new credentials via the new API from (1). InternalStage calls storageManager.getRefreshedLocation and doesn't care whether it is operating on an account's internal stage's streaming_ingest subfolder (FDN ingestion) or on a per-session location under the external volume's S3 location.
  4. Remove from InternalStage and InternalStageManager

Note that after this change, InternalStage can / should be renamed, did not want to make this PR even bigger thus haven't done that yet.

New tests for SubscopedTokenExtVolManager are pending. Sending this out for review + to see if any merge gate tests barf.