search

datamill-co / target-snowflake

A Singer.io Target for Snowflake
MIT License
11 stars 25 forks source link

Snowpipe feature #20

Open mced opened 4 years ago

mced commented 4 years ago

I see that target uses a conventional database connection to copy csv files into snowflake table. Do you think this "target" covers snowpipe usage or do you think that fits to another target ?

https://docs.snowflake.com/en/user-guide/data-load-snowpipe-intro.html https://github.com/snowflakedb/snowflake-ingest-python

AlexanderMann commented 3 years ago

Hey @mced, sorry for the long wait to reply. This got burried underneath a number of other notifications on my end.

My gut reaction is that Target-Snowflake is already doing small batching on its end so that it can keep track of state, so I'm not entirely sure how much of a benefit just using Snowpipe would get us.

That being said, I would imagine that re-architecting around Snowpipe, could lead to some long term boosts in performance. As of now, we haven't seen much in the way of performance concerns for Target-Snowflake, aside from tuning the batching and whatnot.

❓ What specifically about Snowpipe has you interested in it as a feature? Is there some additional benefit from its adoption that I'm missing aside from batching help/support?

mced commented 3 years ago

@AlexanderMann I got interested about Snowpipe because that let me feed Snowflake with appended tables while reducing the cost.

Inserting rows directly to Snowflake requires a datawarehouse up and running.

Snowpipe uses gzipped csv files that have been put in a s3 bucket. Thus you don't have to pay the datawarehouse instance during that time.