search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
183 stars 93 forks source link

Snowflake EDW Write Operation Support #55

Open ShawnHouCHN opened 4 years ago

ShawnHouCHN commented 4 years ago

Background

Currently, Spline seems to not support Snowflake which recently has become popular among EDW solutions

Feature

Snowflake Spark connector read/write support https://github.com/snowflakedb/spark-snowflake

radford1 commented 4 years ago

@wajda do you have any thoughts on how to do integration testing for this? I'm not seeing a way to do an embedded snowflake instance to do continuous testing with.

wajda commented 4 years ago

Yep. Looks like it can only be tested manually. @ShawnHouCHN, can you suggest anything?

wajda commented 4 years ago

I had a quick call with a Snowflake representative about a possibility to have a shareable testing account that we (as an open-source project) could use for integration testing purposes for free. Although they don't provide anything like that at the moment (apart from a 30-days trial), they suppose to discuss this topic and get back to us in a week-ish time. Fingers crossed :)

wajda commented 4 years ago

Unfortunately Snowflake refused to provide a free testing account to an open-source community. A time limited trial account isn't an option from a long-term build stability perspective (unless we could automate a new demo account creation as a part of the build process).