Closed jtmcn closed 7 months ago
- can you confirm that
start_scan_time
is available in non-SnowpipeCOPY INTO
statements? Querying Metadata for Staged Files | Snowflake Documentation makes me think so, but want to double check
Yes, the metadata fields are available when the query target is a Snowflake Stage
- can imagine that this might constitute a breaking change for anyone? My gut tells me "no", but maybe someone who has a downstream model with logic that depends on this
_dbt_copied_at
column might see wildly different times?
No, I don't think this constitutes a breaking change. The new value won't be used until the Snowpipe is recreated. There won't be wild differences within the same table. Also, the documentation says the
CURRENT_TIMESTAMP
is evaluated when the load operation is compiled in cloud services rather than when the record is inserted into the table
The existing value for the _dbt_copied_at
field is likely to be incorrect for it's intended downstream purpose.
Description & motivation
resovles: #281
This change uses
metadata$start_scan_time
instead ofcurrent_timestamp
for the_dbt_copied_at
field on Snowflake Snowpipe creation.This is the method recommended in the Snowflake docs
Checklist