MeltanoLabs / tap-snowflake

Other
4 stars 9 forks source link

feat: native Snowflake bulk unload for `BATCH` messages #1

Closed aaronsteers closed 1 year ago

aaronsteers commented 2 years ago

Related:

Spec details:

  1. Use table stages to avoid requiring S3 creds.
  2. Use default encoding of jsonl and gzip to match initial built-in SDK feature spec.
  3. Skip record validation to avoid needing to parse the raw files.
  4. One BATCH message and manifest per COPY command. (Watch COPY command will contain multiple files.)
aaronsteers commented 2 years ago

cc @edgarrmondragon