search

MeltanoLabs / target-snowflake

Singer Target for the Snowflake cloud Data Warehouse
https://hub.meltano.com/loaders/target-snowflake--meltanolabs/
Other
10 stars 24 forks source link

chore: enable overwrites #285

Open steven-luabase opened 3 weeks ago

steven-luabase commented 3 weeks ago

Previously the load_method = overwrite was not working. We had to enable it in the connector.

edgarrmondragon commented 2 weeks ago

@steven-luabase Does this actually work? It seems to be doing nothing upstream in https://github.com/meltano/sdk/blob/4a33701569b57159d8b4fad26b5a13f37722c47c/singer_sdk/connectors/sql.py#L438.

steven-luabase commented 2 weeks ago

Yes it's working for us. We have a few tables that are dropped-loaded daily and based off the _sdc fields it shows rows are extracted/batched/received etc. daily. I'm also looking at the the dropped tables history using this query on the snowflake account usage views:

SELECT table_id,
  table_name,
  table_schema,
  table_catalog,
  created,
  deleted,
  comment
FROM SNOWFLAKE.ACCOUNT_USAGE.TABLES
WHERE table_catalog = 'MY_SCHEMA'
AND deleted IS NOT NULL
ORDER BY deleted desc

And seeing that those target tables are dropped every time the pipeline is run.